| 1 | # Llama Stack |
| 2 | |
| 3 | vLLM is also available via [Llama Stack](https://github.com/llamastack/llama-stack). |
| 4 | |
| 5 | To install Llama Stack, run |
| 6 | |
| 7 | ```bash |
| 8 | pip install llama-stack -q |
| 9 | ``` |
| 10 | |
| 11 | ## Inference using OpenAI-Compatible API |
| 12 | |
| 13 | Then start the Llama Stack server and configure it to point to your vLLM server with the following settings: |
| 14 | |
| 15 | ```yaml |
| 16 | inference: |
| 17 | - provider_id: vllm0 |
| 18 | provider_type: remote::vllm |
| 19 | config: |
| 20 | url: http://127.0.0.1:8000 |
| 21 | ``` |
| 22 | |
| 23 | Please refer to [this guide](https://llama-stack.readthedocs.io/en/latest/providers/inference/remote_vllm.html) for more details on this remote vLLM provider. |
| 24 | |
| 25 | ## Inference using Embedded vLLM |
| 26 | |
| 27 | An [inline provider](https://github.com/llamastack/llama-stack/tree/main/llama_stack/providers/inline/inference) |
| 28 | is also available. This is a sample of configuration using that method: |
| 29 | |
| 30 | ```yaml |
| 31 | inference: |
| 32 | - provider_type: vllm |
| 33 | config: |
| 34 | model: Llama3.1-8B-Instruct |
| 35 | tensor_parallel_size: 4 |
| 36 | ``` |