A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 09:45 UTC
[Misc] add litellm integration (#18320)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
Reid committed
May 18, 2025 at 23:32 UTC
b6a6e7a529d72e50cbe0c0b5360cf890124260e6
2 files changed
+76
docs/source/deployment/frameworks/index.md
+1
@@ -10,6 +10,7 @@ chatbox
10
dify
11
dstack
12
helm
13
+litellm
14
lobe-chat
15
lws
16
modal
docs/source/deployment/frameworks/litellm.md
new
+75
@@ -0,0 +1,75 @@
1
+(deployment-litellm)=
2
+
3
+# LiteLLM
4
+
5
+[LiteLLM](https://github.com/BerriAI/litellm) call all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, Groq etc.]
6
+
7
+LiteLLM manages:
8
+
9
+- Translate inputs to provider's `completion`, `embedding`, and `image_generation` endpoints
10
+- [Consistent output](https://docs.litellm.ai/docs/completion/output), text responses will always be available at `['choices'][0]['message']['content']`
11
+- Retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - [Router](https://docs.litellm.ai/docs/routing)
12
+- Set Budgets & Rate limits per project, api key, model [LiteLLM Proxy Server (LLM Gateway)](https://docs.litellm.ai/docs/simple_proxy)
13
+
14
+And LiteLLM supports all models on VLLM.
15
+
16
+## Prerequisites
17
+
18
+- Setup vLLM and litellm environment
19
+
20
+```console
21
+pip install vllm litellm
22
+```
23
+
24
+## Deploy
25
+
26
+### Chat completion
27
+
28
+- Start the vLLM server with the supported chat completion model, e.g.
29
+
30
+```console
31
+vllm serve qwen/Qwen1.5-0.5B-Chat
32
+```
33
+
34
+- Call it with litellm:
35
+
36
+```python
37
+import litellm
38
+
39
+messages = [{ "content": "Hello, how are you?","role": "user"}]
40
+
41
+# hosted_vllm is prefix key word and necessary
42
+response = litellm.completion(
43
+ model="hosted_vllm/qwen/Qwen1.5-0.5B-Chat", # pass the vllm model name
44
+ messages=messages,
45
+ api_base="http://{your-vllm-server-host}:{your-vllm-server-port}/v1",
46
+ temperature=0.2,
47
+ max_tokens=80)
48
+
49
+print(response)
50
+```
51
+
52
+### Embeddings
53
+
54
+- Start the vLLM server with the supported embedding model, e.g.
55
+
56
+```console
57
+vllm serve BAAI/bge-base-en-v1.5
58
+```
59
+
60
+- Call it with litellm:
61
+
62
+```python
63
+from litellm import embedding
64
+import os
65
+
66
+os.environ["HOSTED_VLLM_API_BASE"] = "http://{your-vllm-server-host}:{your-vllm-server-port}/v1"
67
+
68
+# hosted_vllm is prefix key word and necessary
69
+# pass the vllm model name
70
+embedding = embedding(model="hosted_vllm/BAAI/bge-base-en-v1.5", input=["Hello world"])
71
+
72
+print(embedding)
73
+```
74
+
75
+For details, see the tutorial [Using vLLM in LiteLLM](https://docs.litellm.ai/docs/providers/vllm).