| 1 | # Prometheus and Grafana |
| 2 | |
| 3 | This is a simple example that shows you how to connect vLLM metric logging to the Prometheus/Grafana stack. For this example, we launch Prometheus and Grafana via Docker. You can checkout other methods through [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) websites. |
| 4 | |
| 5 | Install: |
| 6 | |
| 7 | - [`docker`](https://docs.docker.com/engine/install/) |
| 8 | - [`docker compose`](https://docs.docker.com/compose/install/linux/#install-using-the-repository) |
| 9 | |
| 10 | ## Launch |
| 11 | |
| 12 | Prometheus metric logging is enabled by default in the OpenAI-compatible server. Launch via the entrypoint: |
| 13 | |
| 14 | ```bash |
| 15 | vllm serve mistralai/Mistral-7B-v0.1 \ |
| 16 | --max-model-len 2048 |
| 17 | ``` |
| 18 | |
| 19 | Launch Prometheus and Grafana servers with `docker compose`: |
| 20 | |
| 21 | ```bash |
| 22 | docker compose up |
| 23 | ``` |
| 24 | |
| 25 | Submit some sample requests to the server: |
| 26 | |
| 27 | ```bash |
| 28 | wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json |
| 29 | |
| 30 | vllm bench serve \ |
| 31 | --model mistralai/Mistral-7B-v0.1 \ |
| 32 | --tokenizer mistralai/Mistral-7B-v0.1 \ |
| 33 | --endpoint /v1/completions \ |
| 34 | --dataset-name sharegpt \ |
| 35 | --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \ |
| 36 | --request-rate 3.0 |
| 37 | ``` |
| 38 | |
| 39 | Navigating to [`http://localhost:8000/metrics`](http://localhost:8000/metrics) will show the raw Prometheus metrics being exposed by vLLM. |
| 40 | |
| 41 | ## Grafana Dashboard |
| 42 | |
| 43 | Navigate to [`http://localhost:3000`](http://localhost:3000). Log in with the default username (`admin`) and password (`admin`). |
| 44 | |
| 45 | ### Add Prometheus Data Source |
| 46 | |
| 47 | Navigate to [`http://localhost:3000/connections/datasources/new`](http://localhost:3000/connections/datasources/new) and select Prometheus. |
| 48 | |
| 49 | On Prometheus configuration page, we need to add the `Prometheus Server URL` in `Connection`. For this setup, Grafana and Prometheus are running in separate containers, but Docker creates DNS name for each container. You can just use `http://prometheus:9090`. |
| 50 | |
| 51 | Click `Save & Test`. You should get a green check saying "Successfully queried the Prometheus API.". |
| 52 | |
| 53 | ### Import Dashboard |
| 54 | |
| 55 | Navigate to [`http://localhost:3000/dashboard/import`](http://localhost:3000/dashboard/import), upload `grafana.json`, and select the `prometheus` datasource. You should see a screen that looks like the following: |
| 56 | |
| 57 |  |