A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 11:00 UTC
[Misc] add dify integration (#17895)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
Reid committed
May 9, 2025 at 18:42 UTC
ec61ea20a83d56aaad83cb36bf6098625c564801
5 files changed
+57
docs/source/assets/deployment/dify-chat.png
Binary files /dev/null and b/docs/source/assets/deployment/dify-chat.png differ
docs/source/assets/deployment/dify-create-chatbot.png
Binary files /dev/null and b/docs/source/assets/deployment/dify-create-chatbot.png differ
docs/source/assets/deployment/dify-settings.png
Binary files /dev/null and b/docs/source/assets/deployment/dify-settings.png differ
docs/source/deployment/frameworks/dify.md
new
+56
@@ -0,0 +1,56 @@
1
+(deployment-dify)=
2
+
3
+# Dify
4
+
5
+[Dify](https://github.com/langgenius/dify) is an open-source LLM app development platform. Its intuitive interface combines agentic AI workflow, RAG pipeline, agent capabilities, model management, observability features, and more, allowing you to quickly move from prototype to production.
6
+
7
+It supports vLLM as a model provider to efficiently serve large language models.
8
+
9
+This guide walks you through deploying Dify using a vLLM backend.
10
+
11
+## Prerequisites
12
+
13
+- Setup vLLM environment
14
+- Install [Docker](https://docs.docker.com/engine/install/) and [Docker Compose](https://docs.docker.com/compose/install/)
15
+
16
+## Deploy
17
+
18
+- Start the vLLM server with the supported chat completion model, e.g.
19
+
20
+```console
21
+vllm serve Qwen/Qwen1.5-7B-Chat
22
+```
23
+
24
+- Start the Dify server with docker compose ([details](https://github.com/langgenius/dify?tab=readme-ov-file#quick-start)):
25
+
26
+```console
27
+git clone https://github.com/langgenius/dify.git
28
+cd dify
29
+cd docker
30
+cp .env.example .env
31
+docker compose up -d
32
+```
33
+
34
+- Open the browser to access `http://localhost/install`, config the basic login information and login.
35
+
36
+- In the top-right user menu (under the profile icon), go to Settings, then click `Model Provider`, and locate the `vLLM` provider to install it.
37
+
38
+- Fill in the model provider details as follows:
39
+ - **Model Type**: `LLM`
40
+ - **Model Name**: `Qwen/Qwen1.5-7B-Chat`
41
+ - **API Endpoint URL**: `http://{vllm_server_host}:{vllm_server_port}/v1`
42
+ - **Model Name for API Endpoint**: `Qwen/Qwen1.5-7B-Chat`
43
+ - **Completion Mode**: `Completion`
44
+
45
+:::{image} /assets/deployment/dify-settings.png
46
+:::
47
+
48
+- To create a test chatbot, go to `Studio → Chatbot → Create from Blank`, then select Chatbot as the type:
49
+
50
+:::{image} /assets/deployment/dify-create-chatbot.png
51
+:::
52
+
53
+- Click the chatbot you just created to open the chat interface and start interacting with the model:
54
+
55
+:::{image} /assets/deployment/dify-chat.png
56
+:::
docs/source/deployment/frameworks/index.md
+1
@@ -7,6 +7,7 @@ anything-llm
7
bentoml
8
cerebrium
9
chatbox
10
+dify
11
dstack
12
helm
13
lws