A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Docs] Use 1-2-3 list for deploy steps in deployment/frameworks/ (#24633)
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
Michael Yao committed
Sep 11, 2025 at 16:50 UTC
d14c4ebf08c9a6b6c4131eb3021d50da6fb0c212
6 files changed
+98
-90
docs/deployment/frameworks/autogen.md
+7
-9
@@ -4,9 +4,7 @@
4
5
## Prerequisites
6
7
-- Setup vLLM environment
8
-
9
-- Setup [AutoGen](https://microsoft.github.io/autogen/0.2/docs/installation/) environment
7
+Set up the vLLM and [AutoGen](https://microsoft.github.io/autogen/0.2/docs/installation/) environment:
8
9
```bash
10
pip install vllm
@@ -18,14 +16,14 @@ pip install -U "autogen-agentchat" "autogen-ext[openai]"
16
17
## Deploy
18
21
-- Start the vLLM server with the supported chat completion model, e.g.
19
+1. Start the vLLM server with the supported chat completion model, e.g.
20
23
-```bash
24
-python -m vllm.entrypoints.openai.api_server \
25
- --model mistralai/Mistral-7B-Instruct-v0.2
26
-```
21
+ ```bash
22
+ python -m vllm.entrypoints.openai.api_server \
23
+ --model mistralai/Mistral-7B-Instruct-v0.2
24
+ ```
25
28
-- Call it with AutoGen:
26
+1. Call it with AutoGen:
27
28
??? code
29
docs/deployment/frameworks/chatbox.md
+14
-10
@@ -6,27 +6,31 @@ It allows you to deploy a large language model (LLM) server with vLLM as the bac
6
7
## Prerequisites
8
9
-- Setup vLLM environment
9
+Set up the vLLM environment:
10
+
11
+```bash
12
+pip install vllm
13
+```
14
15
## Deploy
16
13
-- Start the vLLM server with the supported chat completion model, e.g.
17
+1. Start the vLLM server with the supported chat completion model, e.g.
18
15
-```bash
16
-vllm serve qwen/Qwen1.5-0.5B-Chat
17
-```
19
+ ```bash
20
+ vllm serve qwen/Qwen1.5-0.5B-Chat
21
+ ```
22
19
-- Download and install [Chatbox desktop](https://chatboxai.app/en#download).
23
+1. Download and install [Chatbox desktop](https://chatboxai.app/en#download).
24
21
-- On the bottom left of settings, Add Custom Provider
25
+1. On the bottom left of settings, Add Custom Provider
26
- API Mode: `OpenAI API Compatible`
27
- Name: vllm
28
- API Host: `http://{vllm server host}:{vllm server port}/v1`
29
- API Path: `/chat/completions`
30
- Model: `qwen/Qwen1.5-0.5B-Chat`
31
28
-
32
+ 
33
30
-- Go to `Just chat`, and start to chat:
34
+1. Go to `Just chat`, and start to chat:
35
32
-
36
+ 
docs/deployment/frameworks/dify.md
+28
-22
@@ -8,44 +8,50 @@ This guide walks you through deploying Dify using a vLLM backend.
8
9
## Prerequisites
10
11
-- Setup vLLM environment
12
-- Install [Docker](https://docs.docker.com/engine/install/) and [Docker Compose](https://docs.docker.com/compose/install/)
11
+Set up the vLLM environment:
12
+
13
+```bash
14
+pip install vllm
15
+```
16
+
17
+And install [Docker](https://docs.docker.com/engine/install/) and [Docker Compose](https://docs.docker.com/compose/install/).
18
19
## Deploy
20
16
-- Start the vLLM server with the supported chat completion model, e.g.
21
+1. Start the vLLM server with the supported chat completion model, e.g.
22
18
-```bash
19
-vllm serve Qwen/Qwen1.5-7B-Chat
20
-```
23
+ ```bash
24
+ vllm serve Qwen/Qwen1.5-7B-Chat
25
+ ```
26
22
-- Start the Dify server with docker compose ([details](https://github.com/langgenius/dify?tab=readme-ov-file#quick-start)):
27
+1. Start the Dify server with docker compose ([details](https://github.com/langgenius/dify?tab=readme-ov-file#quick-start)):
28
24
-```bash
25
-git clone https://github.com/langgenius/dify.git
26
-cd dify
27
-cd docker
28
-cp .env.example .env
29
-docker compose up -d
30
-```
29
+ ```bash
30
+ git clone https://github.com/langgenius/dify.git
31
+ cd dify
32
+ cd docker
33
+ cp .env.example .env
34
+ docker compose up -d
35
+ ```
36
+
37
+1. Open the browser to access `http://localhost/install`, config the basic login information and login.
38
32
-- Open the browser to access `http://localhost/install`, config the basic login information and login.
39
+1. In the top-right user menu (under the profile icon), go to Settings, then click `Model Provider`, and locate the `vLLM` provider to install it.
40
34
-- In the top-right user menu (under the profile icon), go to Settings, then click `Model Provider`, and locate the `vLLM` provider to install it.
41
+1. Fill in the model provider details as follows:
42
36
-- Fill in the model provider details as follows:
43
- **Model Type**: `LLM`
44
- **Model Name**: `Qwen/Qwen1.5-7B-Chat`
45
- **API Endpoint URL**: `http://{vllm_server_host}:{vllm_server_port}/v1`
46
- **Model Name for API Endpoint**: `Qwen/Qwen1.5-7B-Chat`
47
- **Completion Mode**: `Completion`
48
43
-
49
+ 
50
45
-- To create a test chatbot, go to `Studio → Chatbot → Create from Blank`, then select Chatbot as the type:
51
+1. To create a test chatbot, go to `Studio → Chatbot → Create from Blank`, then select Chatbot as the type:
52
47
-
53
+ 
54
49
-- Click the chatbot you just created to open the chat interface and start interacting with the model:
55
+1. Click the chatbot you just created to open the chat interface and start interacting with the model:
56
51
-
57
+ 
docs/deployment/frameworks/haystack.md
+6
-6
@@ -6,7 +6,7 @@ It allows you to deploy a large language model (LLM) server with vLLM as the bac
6
7
## Prerequisites
8
9
-- Setup vLLM and Haystack environment
9
+Set up the vLLM and Haystack environment:
10
11
```bash
12
pip install vllm haystack-ai
@@ -14,13 +14,13 @@ pip install vllm haystack-ai
14
15
## Deploy
16
17
-- Start the vLLM server with the supported chat completion model, e.g.
17
+1. Start the vLLM server with the supported chat completion model, e.g.
18
19
-```bash
20
-vllm serve mistralai/Mistral-7B-Instruct-v0.1
21
-```
19
+ ```bash
20
+ vllm serve mistralai/Mistral-7B-Instruct-v0.1
21
+ ```
22
23
-- Use the `OpenAIGenerator` and `OpenAIChatGenerator` components in Haystack to query the vLLM server.
23
+1. Use the `OpenAIGenerator` and `OpenAIChatGenerator` components in Haystack to query the vLLM server.
24
25
??? code
26
docs/deployment/frameworks/litellm.md
+11
-11
@@ -13,7 +13,7 @@ And LiteLLM supports all models on VLLM.
13
14
## Prerequisites
15
16
-- Setup vLLM and litellm environment
16
+Set up the vLLM and litellm environment:
17
18
```bash
19
pip install vllm litellm
@@ -23,13 +23,13 @@ pip install vllm litellm
23
24
### Chat completion
25
26
-- Start the vLLM server with the supported chat completion model, e.g.
26
+1. Start the vLLM server with the supported chat completion model, e.g.
27
28
-```bash
29
-vllm serve qwen/Qwen1.5-0.5B-Chat
30
-```
28
+ ```bash
29
+ vllm serve qwen/Qwen1.5-0.5B-Chat
30
+ ```
31
32
-- Call it with litellm:
32
+1. Call it with litellm:
33
34
??? code
35
@@ -51,13 +51,13 @@ vllm serve qwen/Qwen1.5-0.5B-Chat
51
52
### Embeddings
53
54
-- Start the vLLM server with the supported embedding model, e.g.
54
+1. Start the vLLM server with the supported embedding model, e.g.
55
56
-```bash
57
-vllm serve BAAI/bge-base-en-v1.5
58
-```
56
+ ```bash
57
+ vllm serve BAAI/bge-base-en-v1.5
58
+ ```
59
60
-- Call it with litellm:
60
+1. Call it with litellm:
61
62
```python
63
from litellm import embedding
docs/deployment/frameworks/retrieval_augmented_generation.md
+32
-32
@@ -11,7 +11,7 @@ Here are the integrations:
11
12
### Prerequisites
13
14
-- Setup vLLM and langchain environment
14
+Set up the vLLM and langchain environment:
15
16
```bash
17
pip install -U vllm \
@@ -22,33 +22,33 @@ pip install -U vllm \
22
23
### Deploy
24
25
-- Start the vLLM server with the supported embedding model, e.g.
25
+1. Start the vLLM server with the supported embedding model, e.g.
26
27
-```bash
28
-# Start embedding service (port 8000)
29
-vllm serve ssmits/Qwen2-7B-Instruct-embed-base
30
-```
27
+ ```bash
28
+ # Start embedding service (port 8000)
29
+ vllm serve ssmits/Qwen2-7B-Instruct-embed-base
30
+ ```
31
32
-- Start the vLLM server with the supported chat completion model, e.g.
32
+1. Start the vLLM server with the supported chat completion model, e.g.
33
34
-```bash
35
-# Start chat service (port 8001)
36
-vllm serve qwen/Qwen1.5-0.5B-Chat --port 8001
37
-```
34
+ ```bash
35
+ # Start chat service (port 8001)
36
+ vllm serve qwen/Qwen1.5-0.5B-Chat --port 8001
37
+ ```
38
39
-- Use the script: <gh-file:examples/online_serving/retrieval_augmented_generation_with_langchain.py>
39
+1. Use the script: <gh-file:examples/online_serving/retrieval_augmented_generation_with_langchain.py>
40
41
-- Run the script
41
+1. Run the script
42
43
-```python
44
-python retrieval_augmented_generation_with_langchain.py
45
-```
43
+ ```python
44
+ python retrieval_augmented_generation_with_langchain.py
45
+ ```
46
47
## vLLM + llamaindex
48
49
### Prerequisites
50
51
-- Setup vLLM and llamaindex environment
51
+Set up the vLLM and llamaindex environment:
52
53
```bash
54
pip install vllm \
@@ -60,24 +60,24 @@ pip install vllm \
60
61
### Deploy
62
63
-- Start the vLLM server with the supported embedding model, e.g.
63
+1. Start the vLLM server with the supported embedding model, e.g.
64
65
-```bash
66
-# Start embedding service (port 8000)
67
-vllm serve ssmits/Qwen2-7B-Instruct-embed-base
68
-```
65
+ ```bash
66
+ # Start embedding service (port 8000)
67
+ vllm serve ssmits/Qwen2-7B-Instruct-embed-base
68
+ ```
69
70
-- Start the vLLM server with the supported chat completion model, e.g.
70
+1. Start the vLLM server with the supported chat completion model, e.g.
71
72
-```bash
73
-# Start chat service (port 8001)
74
-vllm serve qwen/Qwen1.5-0.5B-Chat --port 8001
75
-```
72
+ ```bash
73
+ # Start chat service (port 8001)
74
+ vllm serve qwen/Qwen1.5-0.5B-Chat --port 8001
75
+ ```
76
77
-- Use the script: <gh-file:examples/online_serving/retrieval_augmented_generation_with_llamaindex.py>
77
+1. Use the script: <gh-file:examples/online_serving/retrieval_augmented_generation_with_llamaindex.py>
78
79
-- Run the script
79
+1. Run the script:
80
81
-```python
82
-python retrieval_augmented_generation_with_llamaindex.py
83
-```
81
+ ```python
82
+ python retrieval_augmented_generation_with_llamaindex.py
83
+ ```