A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 23:15 UTC
[Doc] Added cerebrium as Integration option (#5553)
milo157 committed
Jun 18, 2024 at 18:56 UTC
2bd231a7b7787407ccba36f966603578842d03f7
2 files changed
+110
docs/source/serving/deploying_with_cerebrium.rst
new
+109
@@ -0,0 +1,109 @@
1
+.. _deploying_with_cerebrium:
2
+
3
+Deploying with Cerebrium
4
+============================
5
+
6
+.. raw:: html
7
+
8
+ <p align="center">
9
+ <img src="https://i.ibb.co/hHcScTT/Screenshot-2024-06-13-at-10-14-54.png" alt="vLLM_plus_cerebrium"/>
10
+ </p>
11
+
12
+vLLM can be run on a cloud based GPU machine with `Cerebrium <https://www.cerebrium.ai/>`__, a serverless AI infrastructure platform that makes it easier for companies to build and deploy AI based applications.
13
+
14
+To install the Cerebrium client, run:
15
+
16
+.. code-block:: console
17
+
18
+ $ pip install cerebrium
19
+ $ cerebrium login
20
+
21
+Next, create your Cerebrium project, run:
22
+
23
+.. code-block:: console
24
+
25
+ $ cerebrium init vllm-project
26
+
27
+Next, to install the required packages, add the following to your cerebrium.toml:
28
+
29
+.. code-block:: toml
30
+
31
+ [cerebrium.dependencies.pip]
32
+ vllm = "latest"
33
+
34
+Next, let us add our code to handle inference for the LLM of your choice(`mistralai/Mistral-7B-Instruct-v0.1` for this example), add the following code to your main.py`:
35
+
36
+.. code-block:: python
37
+
38
+ from vllm import LLM, SamplingParams
39
+
40
+ llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.1")
41
+
42
+ def run(prompts: list[str], temperature: float = 0.8, top_p: float = 0.95):
43
+
44
+ sampling_params = SamplingParams(temperature=temperature, top_p=top_p)
45
+ outputs = llm.generate(prompts, sampling_params)
46
+
47
+ # Print the outputs.
48
+ results = []
49
+ for output in outputs:
50
+ prompt = output.prompt
51
+ generated_text = output.outputs[0].text
52
+ results.append({"prompt": prompt, "generated_text": generated_text})
53
+
54
+ return {"results": results}
55
+
56
+
57
+Then, run the following code to deploy it to the cloud
58
+
59
+.. code-block:: console
60
+
61
+ $ cerebrium deploy
62
+
63
+If successful, you should be returned a CURL command that you can call inference against. Just remember to end the url with the function name you are calling (in our case /run)
64
+
65
+.. code-block:: python
66
+
67
+ curl -X POST https://api.cortex.cerebrium.ai/v4/p-xxxxxx/vllm/run \
68
+ -H 'Content-Type: application/json' \
69
+ -H 'Authorization: <JWT TOKEN>' \
70
+ --data '{
71
+ "prompts": [
72
+ "Hello, my name is",
73
+ "The president of the United States is",
74
+ "The capital of France is",
75
+ "The future of AI is"
76
+ ]
77
+ }'
78
+
79
+You should get a response like:
80
+
81
+.. code-block:: python
82
+
83
+ {
84
+ "run_id": "52911756-3066-9ae8-bcc9-d9129d1bd262",
85
+ "result": {
86
+ "result": [
87
+ {
88
+ "prompt": "Hello, my name is",
89
+ "generated_text": " Sarah, and I'm a teacher. I teach elementary school students. One of"
90
+ },
91
+ {
92
+ "prompt": "The president of the United States is",
93
+ "generated_text": " elected every four years. This is a democratic system.\n\n5. What"
94
+ },
95
+ {
96
+ "prompt": "The capital of France is",
97
+ "generated_text": " Paris.\n"
98
+ },
99
+ {
100
+ "prompt": "The future of AI is",
101
+ "generated_text": " bright, but it's important to approach it with a balanced and nuanced perspective."
102
+ }
103
+ ]
104
+ },
105
+ "run_time_ms": 152.53663063049316
106
+ }
107
+
108
+You now have an autoscaling endpoint where you only pay for the compute you use!
109
+
docs/source/serving/integrations.rst
+1
@@ -8,6 +8,7 @@ Integrations
8
deploying_with_kserve
9
deploying_with_triton
10
deploying_with_bentoml
11
+ deploying_with_cerebrium
12
deploying_with_lws
13
deploying_with_dstack
14
serving_with_langchain