A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Docs] Revise frameworks/anything-llm.md (#24489)
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
Michael Yao committed
Sep 11, 2025 at 16:50 UTC
85df8afdae72866f5142d674a916ec6a0879b9e8
1 file changed
+34
-22
docs/deployment/frameworks/anything-llm.md
+34
-22
@@ -1,41 +1,53 @@
1
-# Anything LLM
1
+# AnythingLLM
2
3
-[Anything LLM](https://github.com/Mintplex-Labs/anything-llm) is a full-stack application that enables you to turn any document, resource, or piece of content into context that any LLM can use as references during chatting.
3
+[AnythingLLM](https://github.com/Mintplex-Labs/anything-llm) is a full-stack application that enables you to turn any document, resource, or piece of content into context that any LLM can use as references during chatting.
4
5
It allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints.
6
7
## Prerequisites
8
9
-- Setup vLLM environment
9
+Set up the vLLM environment:
10
+
11
+```bash
12
+pip install vllm
13
+```
14
15
## Deploy
16
13
-- Start the vLLM server with the supported chat completion model, e.g.
17
+1. Start the vLLM server with a supported chat-completion model, for example:
18
15
-```bash
16
-vllm serve Qwen/Qwen1.5-32B-Chat-AWQ --max-model-len 4096
17
-```
19
+ ```bash
20
+ vllm serve Qwen/Qwen1.5-32B-Chat-AWQ --max-model-len 4096
21
+ ```
22
+
23
+1. Download and install [AnythingLLM Desktop](https://anythingllm.com/desktop).
24
+
25
+1. Configure the AI provider:
26
+
27
+ - At the bottom, click the 🔧 wrench icon -> **Open settings** -> **AI Providers** -> **LLM**.
28
+ - Enter the following values:
29
+ - LLM Provider: Generic OpenAI
30
+ - Base URL: `http://{vllm server host}:{vllm server port}/v1`
31
+ - Chat Model Name: `Qwen/Qwen1.5-32B-Chat-AWQ`
32
19
-- Download and install [Anything LLM desktop](https://anythingllm.com/desktop).
33
+ 
34
21
-- On the bottom left of open settings, AI Providers --> LLM:
22
- - LLM Provider: Generic OpenAI
23
- - Base URL: http://{vllm server host}:{vllm server port}/v1
24
- - Chat Model Name: `Qwen/Qwen1.5-32B-Chat-AWQ`
35
+1. Create a workspace:
36
26
-
37
+ 1. At the bottom, click the ↺ back icon and back to workspaces.
38
+ 1. Create a workspace (e.g., `vllm`) and start chatting.
39
28
-- Back to home page, New Workspace --> create `vllm` workspace, and start to chat:
40
+ 
41
30
-
42
+1. Add a document.
43
32
-- Click the upload button:
33
- - upload the doc
34
- - select the doc and move to the workspace
35
- - save and embed
44
+ 1. Click the 📎 attachment icon.
45
+ 1. Upload a document.
46
+ 1. Select and move the document into your workspace.
47
+ 1. Save and embed it.
48
37
-
49
+ 
50
39
-- Chat again:
51
+1. Chat using your document as context.
52
41
-
53
+ 