@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 4800339c62
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 08:15 UTC

Add docs on serving with Llama Stack (#10183)

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com> Co-authored-by: Russell Bryant <rbryant@redhat.com>

Yuan Tang committed Nov 11, 2024 at 14:28 UTC 4800339c6287465a128288231ac9dcd94ddf27ba
2 files changed +43
docs/source/serving/integrations.rst
+1
@@ -13,3 +13,4 @@ Integrations
13 deploying_with_dstack
14 serving_with_langchain
15 serving_with_llamaindex
16 + serving_with_llamastack
docs/source/serving/serving_with_llamastack.rst new
+42
@@ -0,0 +1,42 @@
1 +.. _run_on_llamastack:
2 +
3 +Serving with Llama Stack
4 +============================
5 +
6 +vLLM is also available via `Llama Stack <https://github.com/meta-llama/llama-stack>`_ .
7 +
8 +To install Llama Stack, run
9 +
10 +.. code-block:: console
11 +
12 + $ pip install llama-stack -q
13 +
14 +Inference using OpenAI Compatible API
15 +-------------------------------------
16 +
17 +Then start Llama Stack server pointing to your vLLM server with the following configuration:
18 +
19 +.. code-block:: yaml
20 +
21 + inference:
22 + - provider_id: vllm0
23 + provider_type: remote::vllm
24 + config:
25 + url: http://127.0.0.1:8000
26 +
27 +Please refer to `this guide <https://github.com/meta-llama/llama-stack/blob/main/docs/source/getting_started/distributions/self_hosted_distro/remote_vllm.md>`_ for more details on this remote vLLM provider.
28 +
29 +Inference via Embedded vLLM
30 +---------------------------
31 +
32 +An `inline vLLM provider
33 +<https://github.com/meta-llama/llama-stack/tree/main/llama_stack/providers/inline/inference/vllm>`_
34 +is also available. This is a sample of configuration using that method:
35 +
36 +.. code-block:: yaml
37 +
38 + inference
39 + - provider_type: vllm
40 + config:
41 + model: Llama3.1-8B-Instruct
42 + tensor_parallel_size: 4