| 1 | # LlamaIndex |
| 2 | |
| 3 | vLLM is also available via [LlamaIndex](https://github.com/run-llama/llama_index) . |
| 4 | |
| 5 | To install LlamaIndex, run |
| 6 | |
| 7 | ```bash |
| 8 | pip install llama-index-llms-vllm -q |
| 9 | ``` |
| 10 | |
| 11 | To run inference on a single or multiple GPUs, use `Vllm` class from `llamaindex`. |
| 12 | |
| 13 | ```python |
| 14 | from llama_index.llms.vllm import Vllm |
| 15 | |
| 16 | llm = Vllm( |
| 17 | model="microsoft/Orca-2-7b", |
| 18 | tensor_parallel_size=4, |
| 19 | max_new_tokens=100, |
| 20 | vllm_kwargs={"gpu_memory_utilization": 0.5}, |
| 21 | ) |
| 22 | ``` |
| 23 | |
| 24 | Please refer to this [Tutorial](https://docs.llamaindex.ai/en/latest/examples/llm/vllm/) for more details. |