@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 76 lines 2.04 KB
Rendered Raw
1 # LiteLLM
2
3 [LiteLLM](https://github.com/BerriAI/litellm) call all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, Groq etc.]
4
5 LiteLLM manages:
6
7 - Translate inputs to provider's `completion`, `embedding`, and `image_generation` endpoints
8 - [Consistent output](https://docs.litellm.ai/docs/completion/output), text responses will always be available at `['choices'][0]['message']['content']`
9 - Retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - [Router](https://docs.litellm.ai/docs/routing)
10 - Set Budgets & Rate limits per project, api key, model [LiteLLM Proxy Server (LLM Gateway)](https://docs.litellm.ai/docs/simple_proxy)
11
12 And LiteLLM supports all models on VLLM.
13
14 ## Prerequisites
15
16 Set up the vLLM and litellm environment:
17
18 ```bash
19 pip install vllm litellm
20 ```
21
22 ## Deploy
23
24 ### Chat completion
25
26 1. Start the vLLM server with the supported chat completion model, e.g.
27
28 ```bash
29 vllm serve qwen/Qwen1.5-0.5B-Chat
30 ```
31
32 1. Call it with litellm:
33
34 ??? code
35
36 ```python
37 import litellm
38
39 messages = [{"content": "Hello, how are you?", "role": "user"}]
40
41 # hosted_vllm is prefix key word and necessary
42 response = litellm.completion(
43 model="hosted_vllm/qwen/Qwen1.5-0.5B-Chat", # pass the vllm model name
44 messages=messages,
45 api_base="http://{your-vllm-server-host}:{your-vllm-server-port}/v1",
46 temperature=0.2,
47 max_tokens=80,
48 )
49
50 print(response)
51 ```
52
53 ### Embeddings
54
55 1. Start the vLLM server with the supported embedding model, e.g.
56
57 ```bash
58 vllm serve BAAI/bge-base-en-v1.5
59 ```
60
61 1. Call it with litellm:
62
63 ```python
64 from litellm import embedding
65 import os
66
67 os.environ["HOSTED_VLLM_API_BASE"] = "http://{your-vllm-server-host}:{your-vllm-server-port}/v1"
68
69 # hosted_vllm is prefix key word and necessary
70 # pass the vllm model name
71 embedding = embedding(model="hosted_vllm/BAAI/bge-base-en-v1.5", input=["Hello world"])
72
73 print(embedding)
74 ```
75
76 For details, see the tutorial [Using vLLM in LiteLLM](https://docs.litellm.ai/docs/providers/vllm).