A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 08:30 UTC
[Misc] add AutoGen integration (#18712)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Reid committed
May 26, 2025 at 21:56 UTC
0665e29998016488e0836ba1abfdb95943731f05
1 file changed
+83
docs/deployment/frameworks/autogen.md
new
+83
@@ -0,0 +1,83 @@
1
+---
2
+title: AutoGen
3
+---
4
+[](){ #deployment-autogen }
5
+
6
+[AutoGen](https://github.com/microsoft/autogen) is a framework for creating multi-agent AI applications that can act autonomously or work alongside humans.
7
+
8
+## Prerequisites
9
+
10
+- Setup vLLM environment
11
+
12
+- Setup [AutoGen](https://microsoft.github.io/autogen/0.2/docs/installation/) environment
13
+
14
+```console
15
+pip install vllm
16
+
17
+# Install AgentChat and OpenAI client from Extensions
18
+# AutoGen requires Python 3.10 or later.
19
+pip install -U "autogen-agentchat" "autogen-ext[openai]"
20
+```
21
+
22
+## Deploy
23
+
24
+- Start the vLLM server with the supported chat completion model, e.g.
25
+
26
+```console
27
+python -m vllm.entrypoints.openai.api_server \
28
+ --model mistralai/Mistral-7B-Instruct-v0.2
29
+```
30
+
31
+- Call it with AutoGen:
32
+
33
+```python
34
+import asyncio
35
+from autogen_core.models import UserMessage
36
+from autogen_ext.models.openai import OpenAIChatCompletionClient
37
+from autogen_core.models import ModelFamily
38
+
39
+
40
+async def main() -> None:
41
+ # Create a model client
42
+ model_client = OpenAIChatCompletionClient(
43
+ model="mistralai/Mistral-7B-Instruct-v0.2",
44
+ base_url="http://{your-vllm-host-ip}:{your-vllm-host-port}/v1",
45
+ api_key="EMPTY",
46
+ model_info={
47
+ "vision": False,
48
+ "function_calling": False,
49
+ "json_output": False,
50
+ "family": ModelFamily.MISTRAL,
51
+ "structured_output": True,
52
+ },
53
+ )
54
+
55
+ messages = [UserMessage(content="Write a very short story about a dragon.", source="user")]
56
+
57
+ # Create a stream.
58
+ stream = model_client.create_stream(messages=messages)
59
+
60
+ # Iterate over the stream and print the responses.
61
+ print("Streamed responses:")
62
+ async for response in stream:
63
+ if isinstance(response, str):
64
+ # A partial response is a string.
65
+ print(response, flush=True, end="")
66
+ else:
67
+ # The last response is a CreateResult object with the complete message.
68
+ print("\n\n------------\n")
69
+ print("The complete response:", flush=True)
70
+ print(response.content, flush=True)
71
+
72
+ # Close the client when done.
73
+ await model_client.close()
74
+
75
+
76
+asyncio.run(main())
77
+```
78
+
79
+For details, see the tutorial:
80
+
81
+- [Using vLLM in AutoGen](https://microsoft.github.io/autogen/0.2/docs/topics/non-openai-models/local-vllm/)
82
+
83
+- [OpenAI-compatible API examples](https://microsoft.github.io/autogen/stable/reference/python/autogen_ext.models.openai.html#autogen_ext.models.openai.OpenAIChatCompletionClient)