A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 08:15 UTC
Add docs on serving with Llama Stack (#10183)
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com> Co-authored-by: Russell Bryant <rbryant@redhat.com>
Yuan Tang committed
Nov 11, 2024 at 14:28 UTC
4800339c6287465a128288231ac9dcd94ddf27ba
2 files changed
+43
docs/source/serving/integrations.rst
+1
@@ -13,3 +13,4 @@ Integrations
13
deploying_with_dstack
14
serving_with_langchain
15
serving_with_llamaindex
16
+ serving_with_llamastack
docs/source/serving/serving_with_llamastack.rst
new
+42
@@ -0,0 +1,42 @@
1
+.. _run_on_llamastack:
2
+
3
+Serving with Llama Stack
4
+============================
5
+
6
+vLLM is also available via `Llama Stack <https://github.com/meta-llama/llama-stack>`_ .
7
+
8
+To install Llama Stack, run
9
+
10
+.. code-block:: console
11
+
12
+ $ pip install llama-stack -q
13
+
14
+Inference using OpenAI Compatible API
15
+-------------------------------------
16
+
17
+Then start Llama Stack server pointing to your vLLM server with the following configuration:
18
+
19
+.. code-block:: yaml
20
+
21
+ inference:
22
+ - provider_id: vllm0
23
+ provider_type: remote::vllm
24
+ config:
25
+ url: http://127.0.0.1:8000
26
+
27
+Please refer to `this guide <https://github.com/meta-llama/llama-stack/blob/main/docs/source/getting_started/distributions/self_hosted_distro/remote_vllm.md>`_ for more details on this remote vLLM provider.
28
+
29
+Inference via Embedded vLLM
30
+---------------------------
31
+
32
+An `inline vLLM provider
33
+<https://github.com/meta-llama/llama-stack/tree/main/llama_stack/providers/inline/inference/vllm>`_
34
+is also available. This is a sample of configuration using that method:
35
+
36
+.. code-block:: yaml
37
+
38
+ inference
39
+ - provider_type: vllm
40
+ config:
41
+ model: Llama3.1-8B-Instruct
42
+ tensor_parallel_size: 4