@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 83 lines 2.75 KB
Rendered Raw
1 # Retrieval-Augmented Generation
2
3 [Retrieval-augmented generation (RAG)](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) is a technique that enables generative artificial intelligence (Gen AI) models to retrieve and incorporate new information. It modifies interactions with a large language model (LLM) so that the model responds to user queries with reference to a specified set of documents, using this information to supplement information from its pre-existing training data. This allows LLMs to use domain-specific and/or updated information. Use cases include providing chatbot access to internal company data or generating responses based on authoritative sources.
4
5 Here are the integrations:
6
7 - vLLM + [langchain](https://github.com/langchain-ai/langchain) + [milvus](https://github.com/milvus-io/milvus)
8 - vLLM + [llamaindex](https://github.com/run-llama/llama_index) + [milvus](https://github.com/milvus-io/milvus)
9
10 ## vLLM + langchain
11
12 ### Prerequisites
13
14 Set up the vLLM and langchain environment:
15
16 ```bash
17 pip install -U vllm \
18 langchain_milvus langchain_openai \
19 langchain_community beautifulsoup4 \
20 langchain-text-splitters
21 ```
22
23 ### Deploy
24
25 1. Start the vLLM server with the supported embedding model, e.g.
26
27 ```bash
28 # Start embedding service (port 8000)
29 vllm serve ssmits/Qwen2-7B-Instruct-embed-base
30 ```
31
32 1. Start the vLLM server with the supported chat completion model, e.g.
33
34 ```bash
35 # Start chat service (port 8001)
36 vllm serve qwen/Qwen1.5-0.5B-Chat --port 8001
37 ```
38
39 1. Use the script: [examples/applications/rag/retrieval_augmented_generation_with_langchain.py](../../../examples/applications/rag/retrieval_augmented_generation_with_langchain.py)
40
41 1. Run the script
42
43 ```bash
44 python retrieval_augmented_generation_with_langchain.py
45 ```
46
47 ## vLLM + llamaindex
48
49 ### Prerequisites
50
51 Set up the vLLM and llamaindex environment:
52
53 ```bash
54 pip install vllm \
55 llama-index llama-index-readers-web \
56 llama-index-llms-openai-like \
57 llama-index-embeddings-openai-like \
58 llama-index-vector-stores-milvus \
59 ```
60
61 ### Deploy
62
63 1. Start the vLLM server with the supported embedding model, e.g.
64
65 ```bash
66 # Start embedding service (port 8000)
67 vllm serve ssmits/Qwen2-7B-Instruct-embed-base
68 ```
69
70 1. Start the vLLM server with the supported chat completion model, e.g.
71
72 ```bash
73 # Start chat service (port 8001)
74 vllm serve qwen/Qwen1.5-0.5B-Chat --port 8001
75 ```
76
77 1. Use the script: [examples/applications/rag/retrieval_augmented_generation_with_llamaindex.py](../../../examples/applications/rag/retrieval_augmented_generation_with_llamaindex.py)
78
79 1. Run the script:
80
81 ```bash
82 python retrieval_augmented_generation_with_llamaindex.py
83 ```