A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 14:45 UTC
[doc] add open-webui example (#16747)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
Reid committed
Apr 17, 2025 at 18:27 UTC
d8e557b5e5a1873f7a48b17154003b1d3679fb6b
3 files changed
+30
docs/source/assets/deployment/open_webui.png
Binary files /dev/null and b/docs/source/assets/deployment/open_webui.png differ
docs/source/deployment/frameworks/index.md
+1
@@ -9,6 +9,7 @@ dstack
9
helm
10
lws
11
modal
12
+open-webui
13
skypilot
14
triton
15
:::
docs/source/deployment/frameworks/open-webui.md
new
+29
@@ -0,0 +1,29 @@
1
+(deployment-open-webui)=
2
+
3
+# Open WebUI
4
+
5
+1. Install the (Docker)[https://docs.docker.com/engine/install/]
6
+
7
+2. Start the vLLM server with the supported chat completion model, e.g.
8
+
9
+```console
10
+vllm serve qwen/Qwen1.5-0.5B-Chat
11
+```
12
+
13
+1. Start the (Open WebUI)[https://github.com/open-webui/open-webui] docker container (replace the vllm serve host and vllm serve port):
14
+
15
+```console
16
+docker run -d -p 3000:8080 \
17
+--name open-webui \
18
+-v open-webui:/app/backend/data \
19
+-e OPENAI_API_BASE_URL=http://<vllm serve host>:<vllm serve port>/v1 \
20
+--restart always \
21
+ghcr.io/open-webui/open-webui:main
22
+```
23
+
24
+1. Open it in the browser: <http://open-webui-host:3000/>
25
+
26
+On the top of the web page, you can see the model `qwen/Qwen1.5-0.5B-Chat`.
27
+
28
+:::{image} /assets/deployment/open_webui.png
29
+:::