@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 2bd231a7b7
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 23:15 UTC

[Doc] Added cerebrium as Integration option (#5553)

milo157 committed Jun 18, 2024 at 18:56 UTC 2bd231a7b7787407ccba36f966603578842d03f7
2 files changed +110
docs/source/serving/deploying_with_cerebrium.rst new
+109
@@ -0,0 +1,109 @@
1 +.. _deploying_with_cerebrium:
2 +
3 +Deploying with Cerebrium
4 +============================
5 +
6 +.. raw:: html
7 +
8 + <p align="center">
9 + <img src="https://i.ibb.co/hHcScTT/Screenshot-2024-06-13-at-10-14-54.png" alt="vLLM_plus_cerebrium"/>
10 + </p>
11 +
12 +vLLM can be run on a cloud based GPU machine with `Cerebrium <https://www.cerebrium.ai/>`__, a serverless AI infrastructure platform that makes it easier for companies to build and deploy AI based applications.
13 +
14 +To install the Cerebrium client, run:
15 +
16 +.. code-block:: console
17 +
18 + $ pip install cerebrium
19 + $ cerebrium login
20 +
21 +Next, create your Cerebrium project, run:
22 +
23 +.. code-block:: console
24 +
25 + $ cerebrium init vllm-project
26 +
27 +Next, to install the required packages, add the following to your cerebrium.toml:
28 +
29 +.. code-block:: toml
30 +
31 + [cerebrium.dependencies.pip]
32 + vllm = "latest"
33 +
34 +Next, let us add our code to handle inference for the LLM of your choice(`mistralai/Mistral-7B-Instruct-v0.1` for this example), add the following code to your main.py`:
35 +
36 +.. code-block:: python
37 +
38 + from vllm import LLM, SamplingParams
39 +
40 + llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.1")
41 +
42 + def run(prompts: list[str], temperature: float = 0.8, top_p: float = 0.95):
43 +
44 + sampling_params = SamplingParams(temperature=temperature, top_p=top_p)
45 + outputs = llm.generate(prompts, sampling_params)
46 +
47 + # Print the outputs.
48 + results = []
49 + for output in outputs:
50 + prompt = output.prompt
51 + generated_text = output.outputs[0].text
52 + results.append({"prompt": prompt, "generated_text": generated_text})
53 +
54 + return {"results": results}
55 +
56 +
57 +Then, run the following code to deploy it to the cloud
58 +
59 +.. code-block:: console
60 +
61 + $ cerebrium deploy
62 +
63 +If successful, you should be returned a CURL command that you can call inference against. Just remember to end the url with the function name you are calling (in our case /run)
64 +
65 +.. code-block:: python
66 +
67 + curl -X POST https://api.cortex.cerebrium.ai/v4/p-xxxxxx/vllm/run \
68 + -H 'Content-Type: application/json' \
69 + -H 'Authorization: <JWT TOKEN>' \
70 + --data '{
71 + "prompts": [
72 + "Hello, my name is",
73 + "The president of the United States is",
74 + "The capital of France is",
75 + "The future of AI is"
76 + ]
77 + }'
78 +
79 +You should get a response like:
80 +
81 +.. code-block:: python
82 +
83 + {
84 + "run_id": "52911756-3066-9ae8-bcc9-d9129d1bd262",
85 + "result": {
86 + "result": [
87 + {
88 + "prompt": "Hello, my name is",
89 + "generated_text": " Sarah, and I'm a teacher. I teach elementary school students. One of"
90 + },
91 + {
92 + "prompt": "The president of the United States is",
93 + "generated_text": " elected every four years. This is a democratic system.\n\n5. What"
94 + },
95 + {
96 + "prompt": "The capital of France is",
97 + "generated_text": " Paris.\n"
98 + },
99 + {
100 + "prompt": "The future of AI is",
101 + "generated_text": " bright, but it's important to approach it with a balanced and nuanced perspective."
102 + }
103 + ]
104 + },
105 + "run_time_ms": 152.53663063049316
106 + }
107 +
108 +You now have an autoscaling endpoint where you only pay for the compute you use!
109 +
docs/source/serving/integrations.rst
+1
@@ -8,6 +8,7 @@ Integrations
8 deploying_with_kserve
9 deploying_with_triton
10 deploying_with_bentoml
11 + deploying_with_cerebrium
12 deploying_with_lws
13 deploying_with_dstack
14 serving_with_langchain