[Docs] Update stale LMCache examples (#45762)
Signed-off-by: Samuel Shen <slshen@tensormesh.ai>
Samuel Shen committed
Jun 18, 2026 at 20:21 UTC
c9135db27cafb853af5e2cb86c1a0b3c6b5b8c91
8 files changed
+84
-197
docs/deployment/integrations/production-stack.md
+1
-1
@@ -4,7 +4,7 @@ Deploying vLLM on Kubernetes is a scalable and efficient way to serve machine le
4
5
* **Upstream vLLM compatibility** – It wraps around upstream vLLM without modifying its code.
6
* **Ease of use** – Simplified deployment via Helm charts and observability through Grafana dashboards.
7
-* **High performance** – Optimized for LLM workloads with features like multimodel support, model-aware and prefix-aware routing, fast vLLM bootstrapping, and KV cache offloading with [LMCache](https://github.com/LMCache/LMCache), among others.
7
+* **High performance** – Optimized for LLM workloads with features like multimodel support, model-aware and prefix-aware routing, fast vLLM bootstrapping, and KV cache offloading with [LMCache](https://github.com/LMCache/LMCache) (wired up in vLLM via `--kv-offloading-backend lmcache`; see the [LMCache examples](https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/lmcache) and [docs.lmcache.ai](https://docs.lmcache.ai)), among others.
8
9
If you are new to Kubernetes, don't worry: in the vLLM production stack [repo](https://github.com/vllm-project/production-stack), we provide a step-by-step [guide](https://github.com/vllm-project/production-stack/blob/main/tutorials/00-install-kubernetes-env.md) and a [short video](https://www.youtube.com/watch?v=EsTJbQtzj0g) to set up everything and get started in **4 minutes**!
10
docs/features/disagg_prefill.md
+1
-1
@@ -20,7 +20,7 @@ Two main reasons:
20
Now supports 9 types of connectors:
21
22
- **ExampleConnector**: refer to [examples/disaggregated/example_connector/run.sh](../../examples/disaggregated/example_connector/run.sh) for the example usage of ExampleConnector disaggregated prefilling.
23
-- **LMCacheConnectorV1**: refer to [examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh](../../examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh) for the example usage of LMCacheConnectorV1 disaggregated prefilling which uses NIXL as the underlying KV transmission.
23
+- **LMCacheConnectorV1**: refer to [examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh](../../examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh) for the example usage of LMCacheConnectorV1 disaggregated prefilling which uses NIXL as the underlying KV transmission. LMCache also offers a multi-process (MP) mode via `LMCacheMPConnector`, where a standalone `lmcache server` holds the KV cache shared by one or more vLLM instances; see the [LMCache examples](../../examples/disaggregated/lmcache/README.md) and the [LMCache docs](https://docs.lmcache.ai) for setup.
24
- **NixlConnector**: refer to [tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh](../../tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh) for the example usage of NixlConnector disaggregated prefilling which support fully async send/recv. For detailed usage guide, see [NixlConnector Usage Guide](nixl_connector_usage.md). For feature compatibility details, see [NixlConnector Compatibility Matrix](nixl_connector_compatibility.md). You may specify one or multiple NIXL transfer backends, such as:
25
26
```bash
examples/disaggregated/lmcache/README.md
+34
-14
@@ -1,10 +1,38 @@
1
# LMCache Examples
2
3
-This folder demonstrates how to use LMCache for disaggregated prefilling, CPU offloading and KV cache sharing.
3
+This folder demonstrates how to use LMCache with vLLM v1 for KV cache
4
+offloading, disaggregated prefilling, and KV cache sharing.
5
5
-## 1. Disaggregated Prefill in vLLM v1
6
+## Integration modes
7
7
-This example demonstrates how to run LMCache with disaggregated prefill using NIXL on a single node.
8
+LMCache integrates with vLLM v1 in two ways:
9
+
10
+- **In-process mode** (`LMCacheConnectorV1`): LMCache runs inside the vLLM
11
+ process and is configured through environment variables or a YAML config
12
+ file (`LMCACHE_CONFIG_FILE`). This is the simplest way to add single-node
13
+ CPU/disk offloading.
14
+- **Multi-process (MP) mode** (`LMCacheMPConnector`): LMCache runs as a
15
+ standalone server (`lmcache server`) that owns the KV cache storage; one or
16
+ more vLLM instances connect to it. This is the recommended mode for
17
+ distributed KV storage and for sharing KV cache across instances. See the
18
+ [LMCache docs](https://docs.lmcache.ai) for the full MP setup.
19
+
20
+## 1. CPU offload (in-process)
21
+
22
+- `python cpu_offload_lmcache.py` - CPU offloading with `LMCacheConnectorV1`
23
+ for vLLM v1.
24
+
25
+## 2. CPU offload (multi-process)
26
+
27
+- `bash cpu_offload_lmcache_mp.sh` - CPU offloading with `LMCacheMPConnector`,
28
+ using a standalone `lmcache server`. vLLM provides a built-in shortcut for
29
+ this setup via `--kv-offloading-backend lmcache` and
30
+ `--kv-offloading-size <GiB>`.
31
+
32
+## 3. Disaggregated Prefill in vLLM v1
33
+
34
+This example demonstrates how to run LMCache with disaggregated prefill using
35
+NIXL on a single node.
36
37
### Prerequisites
38
@@ -46,15 +74,7 @@ The main script generates several log files:
74
- `decoder.log` - Logs from the decode server
75
- `proxy.log` - Logs from the proxy server
76
49
-## 2. CPU Offload Examples
50
-
51
-- `python cpu_offload_lmcache.py -v v0` - CPU offloading implementation for vLLM v0
52
-- `python cpu_offload_lmcache.py -v v1` - CPU offloading implementation for vLLM v1
53
-
54
-## 3. KV Cache Sharing
55
-
56
-The `kv_cache_sharing_lmcache_v1.py` example demonstrates how to share KV caches between vLLM v1 instances.
57
-
58
-## 4. Disaggregated Prefill in vLLM v0
77
+## 4. KV Cache Sharing
78
60
-The `disaggregated_prefill_lmcache_v0.py` provides an example of how to run disaggregated prefill in vLLM v0.
79
+The `kv_cache_sharing_lmcache_v1.py` example demonstrates how to share KV
80
+caches between vLLM v1 instances through a centralized LMCache server.
examples/disaggregated/lmcache/cpu_offload_lmcache.py
+5
-33
@@ -1,20 +1,8 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
"""
4
-This file demonstrates the example usage of cpu offloading
5
-with LMCache in vLLM v1 or v0.
6
-
7
-Usage:
8
-
9
- Specify vLLM version
10
-
11
- -v v0 : Use LMCacheConnector
12
- model = mistralai/Mistral-7B-Instruct-v0.2
13
- (Includes enable_chunked_prefill = True)
14
-
15
- -v v1 : Use LMCacheConnectorV1 (default)
16
- model = meta-llama/Meta-Llama-3.1-8B-Instruct
17
- (Without enable_chunked_prefill)
4
+This file demonstrates the example usage of CPU offloading
5
+with LMCache in vLLM v1.
6
7
Note that `lmcache` is needed to run this example.
8
Requirements:
@@ -23,7 +11,6 @@ Learn more about LMCache environment setup, please refer to:
11
https://docs.lmcache.ai/getting_started/installation.html
12
"""
13
26
-import argparse
14
import contextlib
15
import os
16
import time
@@ -39,8 +26,6 @@ from vllm.engine.arg_utils import EngineArgs
26
27
def setup_environment_variables():
28
# LMCache-related environment variables
42
- # Use experimental features in LMCache
43
- os.environ["LMCACHE_USE_EXPERIMENTAL"] = "True"
29
# LMCache is set to use 256 tokens per chunk
30
os.environ["LMCACHE_CHUNK_SIZE"] = "256"
31
# Enable local CPU backend in LMCache
@@ -50,9 +35,9 @@ def setup_environment_variables():
35
36
37
@contextlib.contextmanager
53
-def build_llm_with_lmcache(lmcache_connector: str, model: str):
38
+def build_llm_with_lmcache(model: str):
39
ktc = KVTransferConfig(
55
- kv_connector=lmcache_connector,
40
+ kv_connector="LMCacheConnectorV1",
41
kv_role="kv_both",
42
)
43
# Set GPU memory utilization to 0.8 for an A40 GPU with 40GB
@@ -92,23 +77,10 @@ def print_output(
77
print("-" * 50)
78
79
95
-def parse_args():
96
- parser = argparse.ArgumentParser()
97
- parser.add_argument(
98
- "-v",
99
- "--version",
100
- choices=["v0", "v1"],
101
- default="v1",
102
- help="Specify vLLM version (default: v1)",
103
- )
104
- return parser.parse_args()
105
-
106
-
80
def main():
108
- lmcache_connector = "LMCacheConnectorV1"
81
model = "meta-llama/Meta-Llama-3.1-8B-Instruct"
82
setup_environment_variables()
111
- with build_llm_with_lmcache(lmcache_connector, model) as llm:
83
+ with build_llm_with_lmcache(model) as llm:
84
# This example script runs two requests with a shared prefix.
85
# Define the shared prompt and specific prompts
86
shared_prompt = "Hello, how are you?" * 1000
examples/disaggregated/lmcache/cpu_offload_lmcache_mp.sh
new
+43
@@ -0,0 +1,43 @@
1
+#!/bin/bash
2
+# SPDX-License-Identifier: Apache-2.0
3
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
4
+#
5
+# CPU offloading with LMCache in multi-process (MP) mode.
6
+#
7
+# In MP mode, LMCache runs as a standalone server process (`lmcache server`)
8
+# that owns the KV cache storage. One or more vLLM instances connect to it via
9
+# the `LMCacheMPConnector`. This is the recommended way to run LMCache for
10
+# distributed KV storage and for sharing KV cache across vLLM instances.
11
+#
12
+# vLLM ships a built-in shortcut for this setup: pass `--kv-offloading-backend
13
+# lmcache` together with `--kv-offloading-size <GiB>` and vLLM wires up the
14
+# `LMCacheMPConnector` for you (it defaults to the LMCache server at
15
+# tcp://localhost:5555, matching the `lmcache server` default).
16
+#
17
+# Requires `lmcache` to be installed (`pip install lmcache`).
18
+# Learn more: https://docs.lmcache.ai
19
+set -euo pipefail
20
+
21
+MODEL=${MODEL:-meta-llama/Meta-Llama-3.1-8B-Instruct}
22
+
23
+# 1. Launch the standalone LMCache server (binds tcp://localhost:5555 by
24
+# default). `--l1-size-gb` sets the CPU memory budget for the L1 cache.
25
+echo "Starting LMCache server..."
26
+lmcache server --host localhost --port 5555 --l1-size-gb 5 &
27
+LMCACHE_SERVER_PID=$!
28
+trap 'kill $LMCACHE_SERVER_PID 2>/dev/null || true' EXIT
29
+
30
+# 2. Launch vLLM and offload KV cache to the LMCache server.
31
+# The MP connector currently requires the non-hybrid KV cache manager.
32
+echo "Starting vLLM server with LMCache MP offloading..."
33
+vllm serve "$MODEL" \
34
+ --port 8000 \
35
+ --kv-offloading-size 5 \
36
+ --kv-offloading-backend lmcache \
37
+ --disable-hybrid-kv-cache-manager
38
+
39
+# Equivalent explicit configuration (instead of the two flags above):
40
+# --kv-transfer-config \
41
+# '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both",
42
+# "kv_connector_extra_config":{"lmcache.mp.host":"tcp://localhost",
43
+# "lmcache.mp.port":5555}}'
examples/disaggregated/lmcache/disagg_prefill_lmcache_v0.py
deleted
-144
@@ -1,144 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-"""
4
-This file demonstrates the example usage of disaggregated prefilling
5
-with LMCache.
6
-We will launch 2 vllm instances (GPU 0 for prefill and GPU 1 for decode),
7
-and launch an additional LMCache server.
8
-KV cache is transferred in the following manner:
9
-vLLM prefill node -> LMCache server -> vLLM decode node.
10
-
11
-Note that `pip install lmcache` is needed to run this example.
12
-Learn more about LMCache in https://github.com/LMCache/LMCache.
13
-"""
14
-
15
-import os
16
-import subprocess
17
-import time
18
-from multiprocessing import Event, Process
19
-
20
-from lmcache.experimental.cache_engine import LMCacheEngineBuilder
21
-from lmcache.integration.vllm.utils import ENGINE_NAME
22
-
23
-from vllm import LLM, SamplingParams
24
-from vllm.config import KVTransferConfig
25
-
26
-# LMCache-related environment variables
27
-# The port to start LMCache server
28
-port = 8100
29
-# Use experimental features in LMCache
30
-os.environ["LMCACHE_USE_EXPERIMENTAL"] = "True"
31
-# LMCache is set to use 256 tokens per chunk
32
-os.environ["LMCACHE_CHUNK_SIZE"] = "256"
33
-# Disable local CPU backend in LMCache
34
-os.environ["LMCACHE_LOCAL_CPU"] = "False"
35
-# Set local CPU memory buffer limit to 5.0 GB
36
-os.environ["LMCACHE_MAX_LOCAL_CPU_SIZE"] = "5.0"
37
-# Set the remote URL for LMCache server
38
-os.environ["LMCACHE_REMOTE_URL"] = f"lm://localhost:{port}"
39
-# Set the serializer/deserializer between vllm and LMCache server
40
-# `naive` indicates using raw bytes of the tensor without any compression
41
-os.environ["LMCACHE_REMOTE_SERDE"] = "naive"
42
-
43
-prompts = [
44
- "Hello, how are you?" * 1000,
45
-]
46
-
47
-
48
-def run_prefill(prefill_done, prompts):
49
- # We use GPU 0 for prefill node.
50
- os.environ["CUDA_VISIBLE_DEVICES"] = "0"
51
-
52
- sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=1)
53
-
54
- ktc = KVTransferConfig(
55
- kv_connector="LMCacheConnector",
56
- kv_role="kv_producer",
57
- kv_rank=0,
58
- kv_parallel_size=2,
59
- )
60
- # Set GPU memory utilization to 0.8 for an A40 GPU with 40GB
61
- # memory. Reduce the value if your GPU has less memory.
62
- llm = LLM(
63
- model="mistralai/Mistral-7B-Instruct-v0.2",
64
- kv_transfer_config=ktc,
65
- max_model_len=8000,
66
- gpu_memory_utilization=0.8,
67
- enforce_eager=True,
68
- )
69
-
70
- # llm.generate(prompts, sampling_params)
71
- outputs = llm.generate(prompts, sampling_params)
72
- for output in outputs:
73
- generated_text = output.outputs[0].text
74
- print(f"Generated text: {generated_text!r}")
75
- print("Prefill node is finished.")
76
- prefill_done.set()
77
-
78
- # Clean up lmcache backend
79
- LMCacheEngineBuilder.destroy(ENGINE_NAME)
80
-
81
-
82
-def run_decode(prefill_done, prompts, timeout=1):
83
- # We use GPU 1 for decode node.
84
- os.environ["CUDA_VISIBLE_DEVICES"] = "1"
85
-
86
- sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=10)
87
-
88
- ktc = KVTransferConfig(
89
- kv_connector="LMCacheConnector",
90
- kv_role="kv_consumer",
91
- kv_rank=1,
92
- kv_parallel_size=2,
93
- )
94
- # Set GPU memory utilization to 0.8 for an A40 GPU with 40GB
95
- # of memory. Reduce the value if your GPU has less memory.
96
- llm = LLM(
97
- model="mistralai/Mistral-7B-Instruct-v0.2",
98
- kv_transfer_config=ktc,
99
- max_model_len=8000,
100
- gpu_memory_utilization=0.8,
101
- enforce_eager=True,
102
- )
103
-
104
- print("Waiting for prefill node to finish...")
105
- prefill_done.wait()
106
- time.sleep(timeout)
107
-
108
- outputs = llm.generate(prompts, sampling_params)
109
- for output in outputs:
110
- generated_text = output.outputs[0].text
111
- print(f"Generated text: {generated_text!r}")
112
-
113
- # Clean up lmcache backend
114
- LMCacheEngineBuilder.destroy(ENGINE_NAME)
115
-
116
-
117
-def run_lmcache_server(port):
118
- server_proc = subprocess.Popen(
119
- ["python", "-m", "lmcache.experimental.server", "localhost", str(port)]
120
- )
121
- return server_proc
122
-
123
-
124
-def main():
125
- prefill_done = Event()
126
- prefill_process = Process(target=run_prefill, args=(prefill_done, prompts))
127
- decode_process = Process(target=run_decode, args=(prefill_done, prompts))
128
- lmcache_server_process = run_lmcache_server(port)
129
-
130
- # Start prefill node
131
- prefill_process.start()
132
-
133
- # Start decode node
134
- decode_process.start()
135
-
136
- # Clean up the processes
137
- decode_process.join()
138
- prefill_process.terminate()
139
- lmcache_server_process.terminate()
140
- lmcache_server_process.wait()
141
-
142
-
143
-if __name__ == "__main__":
144
- main()
examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh
-2
@@ -30,7 +30,6 @@ if [[ $1 == "prefiller" ]]; then
30
31
UCX_TLS=cuda_ipc,cuda_copy,tcp \
32
LMCACHE_CONFIG_FILE=$prefill_config_file \
33
- LMCACHE_USE_EXPERIMENTAL=True \
33
VLLM_ENABLE_V1_MULTIPROCESSING=1 \
34
VLLM_WORKER_MULTIPROC_METHOD=spawn \
35
CUDA_VISIBLE_DEVICES=0 \
@@ -47,7 +46,6 @@ elif [[ $1 == "decoder" ]]; then
46
47
UCX_TLS=cuda_ipc,cuda_copy,tcp \
48
LMCACHE_CONFIG_FILE=$decode_config_file \
50
- LMCACHE_USE_EXPERIMENTAL=True \
49
VLLM_ENABLE_V1_MULTIPROCESSING=1 \
50
VLLM_WORKER_MULTIPROC_METHOD=spawn \
51
CUDA_VISIBLE_DEVICES=1 \
examples/disaggregated/lmcache/kv_cache_sharing_lmcache_v1.py
-2
@@ -26,8 +26,6 @@ from vllm.config import KVTransferConfig
26
# LMCache-related environment variables
27
# The port to start LMCache server
28
port = 8100
29
-# Use experimental features in LMCache
30
-os.environ["LMCACHE_USE_EXPERIMENTAL"] = "True"
29
# LMCache is set to use 256 tokens per chunk
30
os.environ["LMCACHE_CHUNK_SIZE"] = "256"
31
# Disable local CPU backend in LMCache