A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 09:45 UTC
Improve examples rendering in docs and GitHub (#18203)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
May 15, 2025 at 16:57 UTC
51ff154639a31973e6a33fa6208c2b50a88d62e2
7 files changed
+27
-10
examples/offline_inference/disaggregated-prefill-v1/README.md
new
+9
@@ -0,0 +1,9 @@
1
+# Disaggregated Prefill V1
2
+
3
+This example contains scripts that demonstrate disaggregated prefill in the offline setting of vLLM.
4
+
5
+## Files
6
+
7
+- `run.sh` - A helper script that will run `prefill_example.py` and `decode_example.py` sequentially.
8
+- `prefill_example.py` - A script which performs prefill only, saving the KV state to the `local_storage` directory and the prompts to `output.txt`.
9
+- `decode_example.py` - A script which performs decode only, loading the KV state from the `local_storage` directory and the prompts from `output.txt`.
examples/offline_inference/openai_batch/README.md
renamed
+9
-9
@@ -8,7 +8,7 @@ This is a guide to performing batch inference using the OpenAI batch file format
8
9
The OpenAI batch file format consists of a series of json objects on new lines.
10
11
-[See here for an example file.](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/openai/openai_example_batch.jsonl)
11
+[See here for an example file.](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/openai_batch/openai_example_batch.jsonl)
12
13
Each line represents a separate request. See the [OpenAI package reference](https://platform.openai.com/docs/api-reference/batch/requestInput) for more details.
14
@@ -30,13 +30,13 @@ We currently support `/v1/chat/completions`, `/v1/embeddings`, and `/v1/score` e
30
To follow along with this example, you can download the example batch, or create your own batch file in your working directory.
31
32
```console
33
-wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai/openai_example_batch.jsonl
33
+wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai_batch/openai_example_batch.jsonl
34
```
35
36
Once you've created your batch file it should look like this
37
38
```console
39
-$ cat offline_inference/openai/openai_example_batch.jsonl
39
+$ cat offline_inference/openai_batch/openai_example_batch.jsonl
40
{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}
41
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}
42
```
@@ -48,7 +48,7 @@ The batch running tool is designed to be used from the command line.
48
You can run the batch with the following command, which will write its results to a file called `results.jsonl`
49
50
```console
51
-python -m vllm.entrypoints.openai.run_batch -i offline_inference/openai/openai_example_batch.jsonl -o results.jsonl --model meta-llama/Meta-Llama-3-8B-Instruct
51
+python -m vllm.entrypoints.openai.run_batch -i offline_inference/openai_batch/openai_example_batch.jsonl -o results.jsonl --model meta-llama/Meta-Llama-3-8B-Instruct
52
```
53
54
### Step 3: Check your results
@@ -65,10 +65,10 @@ $ cat results.jsonl
65
66
The batch runner supports remote input and output urls that are accessible via http/https.
67
68
-For example, to run against our example input file located at `https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai/openai_example_batch.jsonl`, you can run
68
+For example, to run against our example input file located at `https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai_batch/openai_example_batch.jsonl`, you can run
69
70
```console
71
-python -m vllm.entrypoints.openai.run_batch -i https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai/openai_example_batch.jsonl -o results.jsonl --model meta-llama/Meta-Llama-3-8B-Instruct
71
+python -m vllm.entrypoints.openai.run_batch -i https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai_batch/openai_example_batch.jsonl -o results.jsonl --model meta-llama/Meta-Llama-3-8B-Instruct
72
```
73
74
## Example 3: Integrating with AWS S3
@@ -89,13 +89,13 @@ To integrate with cloud blob storage, we recommend using presigned urls.
89
To follow along with this example, you can download the example batch, or create your own batch file in your working directory.
90
91
```console
92
-wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai/openai_example_batch.jsonl
92
+wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai_batch/openai_example_batch.jsonl
93
```
94
95
Once you've created your batch file it should look like this
96
97
```console
98
-$ cat offline_inference/openai/openai_example_batch.jsonl
98
+$ cat offline_inference/openai_batch/openai_example_batch.jsonl
99
{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}
100
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}
101
```
@@ -103,7 +103,7 @@ $ cat offline_inference/openai/openai_example_batch.jsonl
103
Now upload your batch file to your S3 bucket.
104
105
```console
106
-aws s3 cp offline_inference/openai/openai_example_batch.jsonl s3://MY_BUCKET/MY_INPUT_FILE.jsonl
106
+aws s3 cp offline_inference/openai_batch/openai_example_batch.jsonl s3://MY_BUCKET/MY_INPUT_FILE.jsonl
107
```
108
109
### Step 2: Generate your presigned urls
examples/offline_inference/openai_batch/openai_example_batch.jsonl
renamed
examples/online_serving/disaggregated_serving/README.md
new
+8
@@ -0,0 +1,8 @@
1
+# Disaggregated Serving
2
+
3
+This example contains scripts that demonstrate the disaggregated serving features of vLLM.
4
+
5
+## Files
6
+
7
+- `disagg_proxy_demo.py` - Demonstrates XpYd (X prefill instances, Y decode instances).
8
+- `kv_events.sh` - Demonstrates KV cache event publishing.
examples/online_serving/disaggregated_serving/disagg_proxy_demo.py
renamed
+1
-1
@@ -4,7 +4,7 @@ This file provides a disaggregated prefilling proxy demo to demonstrate an
4
example usage of XpYd disaggregated prefilling.
5
We can launch multiple vllm instances (2 for prefill and 2 for decode), and
6
launch this proxy demo through:
7
- python3 examples/online_serving/disagg_examples/disagg_proxy_demo.py \
7
+ python3 examples/online_serving/disaggregated_serving/disagg_proxy_demo.py \
8
--model $model_name \
9
--prefill localhost:8100 localhost:8101 \
10
--decode localhost:8200 localhost:8201 \