| 1 | # AutoGen |
| 2 | |
| 3 | [AutoGen](https://github.com/microsoft/autogen) is a framework for creating multi-agent AI applications that can act autonomously or work alongside humans. |
| 4 | |
| 5 | ## Prerequisites |
| 6 | |
| 7 | Set up the vLLM and [AutoGen](https://microsoft.github.io/autogen/0.2/docs/installation/) environment: |
| 8 | |
| 9 | ```bash |
| 10 | pip install vllm |
| 11 | |
| 12 | # Install AgentChat and OpenAI client from Extensions |
| 13 | # AutoGen requires Python 3.10 or later. |
| 14 | pip install -U "autogen-agentchat" "autogen-ext[openai]" |
| 15 | ``` |
| 16 | |
| 17 | ## Deploy |
| 18 | |
| 19 | 1. Start the vLLM server with the supported chat completion model, e.g. |
| 20 | |
| 21 | ```bash |
| 22 | vllm serve mistralai/Mistral-7B-Instruct-v0.2 |
| 23 | ``` |
| 24 | |
| 25 | 1. Call it with AutoGen: |
| 26 | |
| 27 | ??? code |
| 28 | |
| 29 | ```python |
| 30 | import asyncio |
| 31 | from autogen_core.models import UserMessage |
| 32 | from autogen_ext.models.openai import OpenAIChatCompletionClient |
| 33 | from autogen_core.models import ModelFamily |
| 34 | |
| 35 | |
| 36 | async def main() -> None: |
| 37 | # Create a model client |
| 38 | model_client = OpenAIChatCompletionClient( |
| 39 | model="mistralai/Mistral-7B-Instruct-v0.2", |
| 40 | base_url="http://{your-vllm-host-ip}:{your-vllm-host-port}/v1", |
| 41 | api_key="EMPTY", |
| 42 | model_info={ |
| 43 | "vision": False, |
| 44 | "function_calling": False, |
| 45 | "json_output": False, |
| 46 | "family": ModelFamily.MISTRAL, |
| 47 | "structured_output": True, |
| 48 | }, |
| 49 | ) |
| 50 | |
| 51 | messages = [UserMessage(content="Write a very short story about a dragon.", source="user")] |
| 52 | |
| 53 | # Create a stream. |
| 54 | stream = model_client.create_stream(messages=messages) |
| 55 | |
| 56 | # Iterate over the stream and print the responses. |
| 57 | print("Streamed responses:") |
| 58 | async for response in stream: |
| 59 | if isinstance(response, str): |
| 60 | # A partial response is a string. |
| 61 | print(response, flush=True, end="") |
| 62 | else: |
| 63 | # The last response is a CreateResult object with the complete message. |
| 64 | print("\n\n------------\n") |
| 65 | print("The complete response:", flush=True) |
| 66 | print(response.content, flush=True) |
| 67 | |
| 68 | # Close the client when done. |
| 69 | await model_client.close() |
| 70 | |
| 71 | |
| 72 | asyncio.run(main()) |
| 73 | ``` |
| 74 | |
| 75 | For details, see the tutorial: |
| 76 | |
| 77 | - [Using vLLM in AutoGen](https://microsoft.github.io/autogen/0.2/docs/topics/non-openai-models/local-vllm/) |
| 78 | |
| 79 | - [OpenAI-compatible API examples](https://microsoft.github.io/autogen/stable/reference/python/autogen_ext.models.openai.html#autogen_ext.models.openai.OpenAIChatCompletionClient) |