@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 53 lines 1.68 KB
Rendered Raw
1 # AnythingLLM
2
3 [AnythingLLM](https://github.com/Mintplex-Labs/anything-llm) is a full-stack application that enables you to turn any document, resource, or piece of content into context that any LLM can use as references during chatting.
4
5 It allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints.
6
7 ## Prerequisites
8
9 Set up the vLLM environment:
10
11 ```bash
12 pip install vllm
13 ```
14
15 ## Deploy
16
17 1. Start the vLLM server with a supported chat-completion model, for example:
18
19 ```bash
20 vllm serve Qwen/Qwen1.5-32B-Chat-AWQ --max-model-len 4096
21 ```
22
23 1. Download and install [AnythingLLM Desktop](https://anythingllm.com/desktop).
24
25 1. Configure the AI provider:
26
27 - At the bottom, click the 🔧 wrench icon -> **Open settings** -> **AI Providers** -> **LLM**.
28 - Enter the following values:
29 - LLM Provider: Generic OpenAI
30 - Base URL: `http://{vllm server host}:{vllm server port}/v1`
31 - Chat Model Name: `Qwen/Qwen1.5-32B-Chat-AWQ`
32
33 ![set AI providers](../../assets/deployment/anything-llm-provider.png)
34
35 1. Create a workspace:
36
37 1. At the bottom, click the ↺ back icon and back to workspaces.
38 1. Create a workspace (e.g., `vllm`) and start chatting.
39
40 ![create a workspace](../../assets/deployment/anything-llm-chat-without-doc.png)
41
42 1. Add a document.
43
44 1. Click the 📎 attachment icon.
45 1. Upload a document.
46 1. Select and move the document into your workspace.
47 1. Save and embed it.
48
49 ![add a document](../../assets/deployment/anything-llm-upload-doc.png)
50
51 1. Chat using your document as context.
52
53 ![chat with your context](../../assets/deployment/anything-llm-chat-with-doc.png)