A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 11:00 UTC
[Misc] add chatbox integration (#17828)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
Reid committed
May 8, 2025 at 18:05 UTC
53d0cb742366a00acd259beb786ac5d8224bc641
4 files changed
+37
docs/source/assets/deployment/chatbox-chat.png
Binary files /dev/null and b/docs/source/assets/deployment/chatbox-chat.png differ
docs/source/assets/deployment/chatbox-settings.png
Binary files /dev/null and b/docs/source/assets/deployment/chatbox-settings.png differ
docs/source/deployment/frameworks/chatbox.md
new
+36
@@ -0,0 +1,36 @@
1
+(deployment-chatbox)=
2
+
3
+# Chatbox
4
+
5
+[Chatbox](https://github.com/chatboxai/chatbox) is a desktop client for LLMs, available on Windows, Mac, Linux.
6
+
7
+It allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints.
8
+
9
+## Prerequisites
10
+
11
+- Setup vLLM environment
12
+
13
+## Deploy
14
+
15
+- Start the vLLM server with the supported chat completion model, e.g.
16
+
17
+```console
18
+vllm serve qwen/Qwen1.5-0.5B-Chat
19
+```
20
+
21
+- Download and install [Chatbox desktop](https://chatboxai.app/en#download).
22
+
23
+- On the bottom left of settings, Add Custom Provider
24
+ - API Mode: `OpenAI API Compatible`
25
+ - Name: vllm
26
+ - API Host: `http://{vllm server host}:{vllm server port}/v1`
27
+ - API Path: `/chat/completions`
28
+ - Model: `qwen/Qwen1.5-0.5B-Chat`
29
+
30
+:::{image} /assets/deployment/chatbox-settings.png
31
+:::
32
+
33
+- Go to `Just chat`, and start to chat:
34
+
35
+:::{image} /assets/deployment/chatbox-chat.png
36
+:::
docs/source/deployment/frameworks/index.md
+1
@@ -6,6 +6,7 @@
6
anything-llm
7
bentoml
8
cerebrium
9
+chatbox
10
dstack
11
helm
12
lws