| 1 | # Cerebrium |
| 2 | |
| 3 | <p align="center"> |
| 4 | <img src="https://i.ibb.co/hHcScTT/Screenshot-2024-06-13-at-10-14-54.png" alt="vLLM_plus_cerebrium"/> |
| 5 | </p> |
| 6 | |
| 7 | vLLM can be run on a cloud based GPU machine with [Cerebrium](https://www.cerebrium.ai/), a serverless AI infrastructure platform that makes it easier for companies to build and deploy AI based applications. |
| 8 | |
| 9 | To install the Cerebrium client, run: |
| 10 | |
| 11 | ```bash |
| 12 | pip install cerebrium |
| 13 | cerebrium login |
| 14 | ``` |
| 15 | |
| 16 | Next, create your Cerebrium project, run: |
| 17 | |
| 18 | ```bash |
| 19 | cerebrium init vllm-project |
| 20 | ``` |
| 21 | |
| 22 | Next, to install the required packages, add the following to your cerebrium.toml: |
| 23 | |
| 24 | ```toml |
| 25 | [cerebrium.deployment] |
| 26 | docker_base_image_url = "nvidia/cuda:12.1.1-runtime-ubuntu22.04" |
| 27 | |
| 28 | [cerebrium.dependencies.pip] |
| 29 | vllm = "latest" |
| 30 | ``` |
| 31 | |
| 32 | Next, let us add our code to handle inference for the LLM of your choice (`mistralai/Mistral-7B-Instruct-v0.1` for this example), add the following code to your `main.py`: |
| 33 | |
| 34 | ??? code |
| 35 | |
| 36 | ```python |
| 37 | from vllm import LLM, SamplingParams |
| 38 | |
| 39 | llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.1") |
| 40 | |
| 41 | def run(prompts: list[str], temperature: float = 0.8, top_p: float = 0.95): |
| 42 | |
| 43 | sampling_params = SamplingParams(temperature=temperature, top_p=top_p) |
| 44 | outputs = llm.generate(prompts, sampling_params) |
| 45 | |
| 46 | # Print the outputs. |
| 47 | results = [] |
| 48 | for output in outputs: |
| 49 | prompt = output.prompt |
| 50 | generated_text = output.outputs[0].text |
| 51 | results.append({"prompt": prompt, "generated_text": generated_text}) |
| 52 | |
| 53 | return {"results": results} |
| 54 | ``` |
| 55 | |
| 56 | Then, run the following code to deploy it to the cloud: |
| 57 | |
| 58 | ```bash |
| 59 | cerebrium deploy |
| 60 | ``` |
| 61 | |
| 62 | If successful, you should be returned a CURL command that you can call inference against. Just remember to end the url with the function name you are calling (in our case `/run`) |
| 63 | |
| 64 | ??? console "Command" |
| 65 | |
| 66 | ```bash |
| 67 | curl -X POST https://api.cortex.cerebrium.ai/v4/p-xxxxxx/vllm/run \ |
| 68 | -H 'Content-Type: application/json' \ |
| 69 | -H 'Authorization: <JWT TOKEN>' \ |
| 70 | --data '{ |
| 71 | "prompts": [ |
| 72 | "Hello, my name is", |
| 73 | "The president of the United States is", |
| 74 | "The capital of France is", |
| 75 | "The future of AI is" |
| 76 | ] |
| 77 | }' |
| 78 | ``` |
| 79 | |
| 80 | You should get a response like: |
| 81 | |
| 82 | ??? console "Response" |
| 83 | |
| 84 | ```json |
| 85 | { |
| 86 | "run_id": "52911756-3066-9ae8-bcc9-d9129d1bd262", |
| 87 | "result": { |
| 88 | "result": [ |
| 89 | { |
| 90 | "prompt": "Hello, my name is", |
| 91 | "generated_text": " Sarah, and I'm a teacher. I teach elementary school students. One of" |
| 92 | }, |
| 93 | { |
| 94 | "prompt": "The president of the United States is", |
| 95 | "generated_text": " elected every four years. This is a democratic system.\n\n5. What" |
| 96 | }, |
| 97 | { |
| 98 | "prompt": "The capital of France is", |
| 99 | "generated_text": " Paris.\n" |
| 100 | }, |
| 101 | { |
| 102 | "prompt": "The future of AI is", |
| 103 | "generated_text": " bright, but it's important to approach it with a balanced and nuanced perspective." |
| 104 | } |
| 105 | ] |
| 106 | }, |
| 107 | "run_time_ms": 152.53663063049316 |
| 108 | } |
| 109 | ``` |
| 110 | |
| 111 | You now have an autoscaling endpoint where you only pay for the compute you use! |