@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 22b39e11f2
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 18:15 UTC

llama_index serving integration documentation (#6973)

Co-authored-by: pavanmantha <pavan.mantha@thevaslabs.io>

Kameshwara Pavan Kumar Mantha committed Aug 15, 2024 at 04:08 UTC 22b39e11f2eca7dd70bcef3760cd5da149412d00
2 files changed +28
docs/source/serving/integrations.rst
+1
@@ -12,3 +12,4 @@ Integrations
12 deploying_with_lws
13 deploying_with_dstack
14 serving_with_langchain
15 + serving_with_llamaindex
docs/source/serving/serving_with_llamaindex.rst new
+27
@@ -0,0 +1,27 @@
1 +.. _run_on_llamaindex:
2 +
3 +Serving with llama_index
4 +============================
5 +
6 +vLLM is also available via `llama_index <https://github.com/run-llama/llama_index>`_ .
7 +
8 +To install llamaindex, run
9 +
10 +.. code-block:: console
11 +
12 + $ pip install llama-index-llms-vllm -q
13 +
14 +To run inference on a single or multiple GPUs, use ``Vllm`` class from ``llamaindex``.
15 +
16 +.. code-block:: python
17 +
18 + from llama_index.llms.vllm import Vllm
19 +
20 + llm = Vllm(
21 + model="microsoft/Orca-2-7b",
22 + tensor_parallel_size=4,
23 + max_new_tokens=100,
24 + vllm_kwargs={"swap_space": 1, "gpu_memory_utilization": 0.5},
25 + )
26 +
27 +Please refer to this `Tutorial <https://docs.llamaindex.ai/en/latest/examples/llm/vllm/>`_ for more details.