A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 18:15 UTC
llama_index serving integration documentation (#6973)
Co-authored-by: pavanmantha <pavan.mantha@thevaslabs.io>
Kameshwara Pavan Kumar Mantha committed
Aug 15, 2024 at 04:08 UTC
22b39e11f2eca7dd70bcef3760cd5da149412d00
2 files changed
+28
docs/source/serving/integrations.rst
+1
@@ -12,3 +12,4 @@ Integrations
12
deploying_with_lws
13
deploying_with_dstack
14
serving_with_langchain
15
+ serving_with_llamaindex
docs/source/serving/serving_with_llamaindex.rst
new
+27
@@ -0,0 +1,27 @@
1
+.. _run_on_llamaindex:
2
+
3
+Serving with llama_index
4
+============================
5
+
6
+vLLM is also available via `llama_index <https://github.com/run-llama/llama_index>`_ .
7
+
8
+To install llamaindex, run
9
+
10
+.. code-block:: console
11
+
12
+ $ pip install llama-index-llms-vllm -q
13
+
14
+To run inference on a single or multiple GPUs, use ``Vllm`` class from ``llamaindex``.
15
+
16
+.. code-block:: python
17
+
18
+ from llama_index.llms.vllm import Vllm
19
+
20
+ llm = Vllm(
21
+ model="microsoft/Orca-2-7b",
22
+ tensor_parallel_size=4,
23
+ max_new_tokens=100,
24
+ vllm_kwargs={"swap_space": 1, "gpu_memory_utilization": 0.5},
25
+ )
26
+
27
+Please refer to this `Tutorial <https://docs.llamaindex.ai/en/latest/examples/llm/vllm/>`_ for more details.