@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / de3e53a75b
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 14, 05:30 UTC

feat: Add Grafana and Perces monitoring dashboards for vLLM (#23498)

liangwen12year committed Sep 16, 2025 at 08:53 UTC de3e53a75ba9f31f446926911b7c44561af3b2ee
7 files changed +3515
examples/online_serving/dashboards/README.md new
+87
@@ -0,0 +1,87 @@
1 +# Monitoring Dashboards
2 +
3 +This directory contains monitoring dashboard configurations for vLLM, providing
4 +comprehensive observability for your vLLM deployments.
5 +
6 +## Dashboard Platforms
7 +
8 +We provide dashboards for two popular observability platforms:
9 +
10 +- **[Grafana](https://grafana.com)**
11 +- **[Perses](https://perses.dev)**
12 +
13 +## Dashboard Format Approach
14 +
15 +All dashboards are provided in **native formats** that work across different
16 +deployment methods:
17 +
18 +### Grafana (JSON)
19 +
20 +- ✅ Works with any Grafana instance (cloud, self-hosted, Docker)
21 +- ✅ Direct import via Grafana UI or API
22 +- ✅ Can be wrapped in Kubernetes operators when needed
23 +- ✅ No vendor lock-in or deployment dependencies
24 +
25 +### Perses (YAML)
26 +
27 +- ✅ Works with standalone Perses instances
28 +- ✅ Compatible with Perses API and CLI
29 +- ✅ Supports Dashboard-as-Code workflows
30 +- ✅ Can be wrapped in Kubernetes operators when needed
31 +
32 +## Dashboard Contents
33 +
34 +Both platforms provide equivalent monitoring capabilities:
35 +
36 +| Dashboard | Description |
37 +|-----------|-------------|
38 +| **Performance Statistics** | Tracks latency, throughput, and performance metrics |
39 +| **Query Statistics** | Monitors request volume, query performance, and KPIs |
40 +
41 +## Quick Start
42 +
43 +First, navigate to this example's directory:
44 +
45 +```bash
46 +cd examples/online_serving/dashboards
47 +```
48 +
49 +### Grafana
50 +
51 +Import the JSON directly into the Grafana UI, or use the API:
52 +
53 +```bash
54 +curl -X POST http://grafana/api/dashboards/db \
55 + -H "Content-Type: application/json" \
56 + -d @grafana/performance_statistics.json
57 +```
58 +
59 +### Perses
60 +
61 +Import via the Perses CLI:
62 +
63 +```bash
64 +percli apply -f perses/performance_statistics.yaml
65 +```
66 +
67 +## Requirements
68 +
69 +- **Prometheus** metrics from your vLLM deployment
70 +- **Data source** configured in your monitoring platform
71 +- **vLLM metrics** enabled and accessible
72 +
73 +## Platform-Specific Documentation
74 +
75 +For detailed deployment instructions and platform-specific options, see:
76 +
77 +- **[Grafana Documentation](./grafana)** - JSON dashboards, operator usage, manual import
78 +- **[Perses Documentation](./perses)** - YAML specs, CLI usage, operator wrapping
79 +
80 +## Contributing
81 +
82 +When adding new dashboards, please:
83 +
84 +1. Provide native formats (JSON for Grafana, YAML specs for Perses)
85 +2. Update platform-specific README files
86 +3. Ensure dashboards work across deployment methods
87 +4. Test with the latest platform versions
examples/online_serving/dashboards/grafana/README.md new
+59
@@ -0,0 +1,59 @@
1 +# Grafana Dashboards for vLLM Monitoring
2 +
3 +This directory contains Grafana dashboard configurations (as JSON) designed to monitor
4 +vLLM performance and metrics.
5 +
6 +## Requirements
7 +
8 +- Grafana 8.0+
9 +- Prometheus data source configured in Grafana
10 +- vLLM deployment with Prometheus metrics enabled
11 +
12 +## Dashboard Descriptions
13 +
14 +- **[performance_statistics.json](./performance_statistics.json)**: Tracks performance metrics including latency and
15 + throughput for your vLLM service.
16 +- **[query_statistics.json](./query_statistics.json)**: Tracks query performance, request volume, and key
17 + performance indicators for your vLLM service.
18 +
19 +## Deployment Options
20 +
21 +### Manual Import (Recommended)
22 +
23 +The easiest way to use these dashboards is to manually import the JSON configurations
24 +directly into your Grafana instance:
25 +
26 +1. Navigate to your Grafana instance
27 +2. Click the '+' icon in the sidebar
28 +3. Select 'Import'
29 +4. Copy and paste the JSON content from the dashboard files, or upload the JSON files
30 + directly
31 +
32 +### Grafana Operator
33 +
34 +If you're using the [Grafana Operator](https://github.com/grafana-operator/grafana-operator)
35 +in Kubernetes, you can wrap these JSON configurations in a `GrafanaDashboard` custom
36 +resource:
37 +
38 +```yaml
39 +# Note: Adjust the instanceSelector to match your Grafana instance's labels
40 +# You can check with: kubectl get grafana -o yaml
41 +apiVersion: grafana.integreatly.org/v1beta1
42 +kind: GrafanaDashboard
43 +metadata:
44 + name: vllm-performance-dashboard
45 +spec:
46 + instanceSelector:
47 + matchLabels:
48 + dashboards: grafana # Adjust to match your Grafana instance labels
49 + folder: "vLLM Monitoring"
50 + json: |
51 + # Replace this comment with the complete JSON content from
52 + # performance_statistics.json - The JSON should start with { and end with }
53 +```
54 +
55 +Then apply to your cluster:
56 +
57 +```bash
58 +kubectl apply -f your-dashboard.yaml -n <namespace>
59 +```
examples/online_serving/dashboards/grafana/performance_statistics.json new
+1405
@@ -0,0 +1,1405 @@
1 +{
2 + "annotations": {
3 + "list": [
4 + {
5 + "builtIn": 1,
6 + "datasource": {
7 + "type": "grafana",
8 + "uid": "-- Grafana --"
9 + },
10 + "enable": true,
11 + "hide": true,
12 + "iconColor": "rgba(0, 211, 255, 1)",
13 + "name": "Annotations & Alerts",
14 + "type": "dashboard"
15 + }
16 + ]
17 + },
18 + "editable": true,
19 + "fiscalYearStartMonth": 0,
20 + "graphTooltip": 0,
21 + "id": 26,
22 + "links": [],
23 + "panels": [
24 + {
25 + "collapsed": false,
26 + "gridPos": {
27 + "h": 1,
28 + "w": 24,
29 + "x": 0,
30 + "y": 0
31 + },
32 + "id": 9,
33 + "panels": [],
34 + "title": "Graph: E2E latency over time ",
35 + "type": "row"
36 + },
37 + {
38 + "datasource": {
39 + "type": "prometheus",
40 + "uid": "${DS_PROMETHEUS}"
41 + },
42 + "description": "End-to-End latency of requests, showing average and key percentiles over time.",
43 + "fieldConfig": {
44 + "defaults": {
45 + "color": {
46 + "mode": "palette-classic"
47 + },
48 + "custom": {
49 + "axisBorderShow": false,
50 + "axisCenteredZero": false,
51 + "axisColorMode": "text",
52 + "axisLabel": "Latency",
53 + "axisPlacement": "auto",
54 + "barAlignment": 0,
55 + "barWidthFactor": 0.6,
56 + "drawStyle": "line",
57 + "fillOpacity": 18,
58 + "gradientMode": "none",
59 + "hideFrom": {
60 + "legend": false,
61 + "tooltip": false,
62 + "viz": false
63 + },
64 + "insertNulls": false,
65 + "lineInterpolation": "linear",
66 + "lineWidth": 1,
67 + "pointSize": 5,
68 + "scaleDistribution": {
69 + "type": "linear"
70 + },
71 + "showPoints": "auto",
72 + "spanNulls": true,
73 + "stacking": {
74 + "group": "A",
75 + "mode": "none"
76 + },
77 + "thresholdsStyle": {
78 + "mode": "off"
79 + }
80 + },
81 + "decimals": 2,
82 + "mappings": [],
83 + "thresholds": {
84 + "mode": "absolute",
85 + "steps": [
86 + {
87 + "color": "green",
88 + "value": null
89 + },
90 + {
91 + "color": "red",
92 + "value": 80
93 + }
94 + ]
95 + },
96 + "unit": "s"
97 + },
98 + "overrides": []
99 + },
100 + "gridPos": {
101 + "h": 8,
102 + "w": 12,
103 + "x": 0,
104 + "y": 1
105 + },
106 + "id": 1,
107 + "options": {
108 + "legend": {
109 + "calcs": [],
110 + "displayMode": "list",
111 + "placement": "bottom",
112 + "showLegend": true
113 + },
114 + "tooltip": {
115 + "mode": "single",
116 + "sort": "none"
117 + }
118 + },
119 + "pluginVersion": "11.3.0",
120 + "targets": [
121 + {
122 + "datasource": {
123 + "type": "prometheus",
124 + "uid": "${DS_PROMETHEUS}"
125 + },
126 + "editorMode": "code",
127 + "expr": "rate(vllm:e2e_request_latency_seconds_sum[$__interval]) / rate(vllm:e2e_request_latency_seconds_count[$__interval])",
128 + "format": "table",
129 + "legendFormat": "E2E Latency",
130 + "range": true,
131 + "refId": "A"
132 + }
133 + ],
134 + "title": "E2E Latency over Time",
135 + "type": "timeseries"
136 + },
137 + {
138 + "datasource": {
139 + "type": "prometheus",
140 + "uid": "${DS_PROMETHEUS}"
141 + },
142 + "description": "99th percentile of End-to-End request latency over the selected time range.",
143 + "fieldConfig": {
144 + "defaults": {
145 + "color": {
146 + "mode": "thresholds"
147 + },
148 + "decimals": 2,
149 + "displayName": "P99",
150 + "mappings": [],
151 + "thresholds": {
152 + "mode": "absolute",
153 + "steps": [
154 + {
155 + "color": "green",
156 + "value": null
157 + },
158 + {
159 + "color": "red",
160 + "value": 80
161 + }
162 + ]
163 + },
164 + "unit": "s"
165 + },
166 + "overrides": []
167 + },
168 + "gridPos": {
169 + "h": 4,
170 + "w": 6,
171 + "x": 12,
172 + "y": 1
173 + },
174 + "id": 5,
175 + "options": {
176 + "colorMode": "value",
177 + "graphMode": "area",
178 + "justifyMode": "auto",
179 + "orientation": "auto",
180 + "percentChangeColorMode": "standard",
181 + "reduceOptions": {
182 + "calcs": [
183 + "lastNotNull"
184 + ],
185 + "fields": "",
186 + "values": false
187 + },
188 + "showPercentChange": false,
189 + "textMode": "auto",
190 + "wideLayout": true
191 + },
192 + "pluginVersion": "11.3.0",
193 + "targets": [
194 + {
195 + "editorMode": "code",
196 + "expr": "histogram_quantile(0.99, sum by(le) (rate(vllm:e2e_request_latency_seconds_bucket[$__range])))",
197 + "legendFormat": "__auto",
198 + "range": true,
199 + "refId": "A"
200 + }
201 + ],
202 + "title": "E2E Latency (P99)",
203 + "type": "stat"
204 + },
205 + {
206 + "datasource": {
207 + "type": "prometheus",
208 + "uid": "${DS_PROMETHEUS}"
209 + },
210 + "description": "90th percentile of End-to-End request latency over the selected time range.",
211 + "fieldConfig": {
212 + "defaults": {
213 + "color": {
214 + "mode": "thresholds"
215 + },
216 + "decimals": 2,
217 + "displayName": "P90",
218 + "mappings": [],
219 + "thresholds": {
220 + "mode": "absolute",
221 + "steps": [
222 + {
223 + "color": "green",
224 + "value": null
225 + },
226 + {
227 + "color": "red",
228 + "value": 80
229 + }
230 + ]
231 + },
232 + "unit": "s"
233 + },
234 + "overrides": []
235 + },
236 + "gridPos": {
237 + "h": 4,
238 + "w": 6,
239 + "x": 18,
240 + "y": 1
241 + },
242 + "id": 4,
243 + "options": {
244 + "colorMode": "value",
245 + "graphMode": "area",
246 + "justifyMode": "auto",
247 + "orientation": "auto",
248 + "percentChangeColorMode": "standard",
249 + "reduceOptions": {
250 + "calcs": [
251 + "lastNotNull"
252 + ],
253 + "fields": "",
254 + "values": false
255 + },
256 + "showPercentChange": false,
257 + "textMode": "auto",
258 + "wideLayout": true
259 + },
260 + "pluginVersion": "11.3.0",
261 + "targets": [
262 + {
263 + "editorMode": "code",
264 + "expr": "histogram_quantile(0.90, sum by(le) (rate(vllm:e2e_request_latency_seconds_bucket[$__range])))",
265 + "legendFormat": "__auto",
266 + "range": true,
267 + "refId": "A"
268 + }
269 + ],
270 + "title": "E2E Latency (P90)",
271 + "type": "stat"
272 + },
273 + {
274 + "datasource": {
275 + "type": "prometheus",
276 + "uid": "${DS_PROMETHEUS}"
277 + },
278 + "description": "Average End-to-End request latency over the selected time range.",
279 + "fieldConfig": {
280 + "defaults": {
281 + "color": {
282 + "mode": "thresholds"
283 + },
284 + "decimals": 2,
285 + "displayName": "Average",
286 + "mappings": [],
287 + "thresholds": {
288 + "mode": "absolute",
289 + "steps": [
290 + {
291 + "color": "green",
292 + "value": null
293 + },
294 + {
295 + "color": "red",
296 + "value": 80
297 + }
298 + ]
299 + },
300 + "unit": "s"
301 + },
302 + "overrides": []
303 + },
304 + "gridPos": {
305 + "h": 4,
306 + "w": 6,
307 + "x": 12,
308 + "y": 5
309 + },
310 + "id": 2,
311 + "options": {
312 + "colorMode": "value",
313 + "graphMode": "area",
314 + "justifyMode": "auto",
315 + "orientation": "auto",
316 + "percentChangeColorMode": "standard",
317 + "reduceOptions": {
318 + "calcs": [
319 + "lastNotNull"
320 + ],
321 + "fields": "",
322 + "values": false
323 + },
324 + "showPercentChange": false,
325 + "textMode": "auto",
326 + "wideLayout": true
327 + },
328 + "pluginVersion": "11.3.0",
329 + "targets": [
330 + {
331 + "editorMode": "code",
332 + "expr": "(sum(increase(vllm:e2e_request_latency_seconds_sum[$__range])) / sum(increase(vllm:e2e_request_latency_seconds_count[$__range])))",
333 + "legendFormat": "Average E2E Latency",
334 + "range": true,
335 + "refId": "A"
336 + }
337 + ],
338 + "title": "E2E Latency (Avg)",
339 + "type": "stat"
340 + },
341 + {
342 + "datasource": {
343 + "type": "prometheus",
344 + "uid": "${DS_PROMETHEUS}"
345 + },
346 + "description": "50th percentile (median) of End-to-End request latency over the selected time range.",
347 + "fieldConfig": {
348 + "defaults": {
349 + "color": {
350 + "mode": "thresholds"
351 + },
352 + "decimals": 2,
353 + "displayName": "P50",
354 + "mappings": [],
355 + "thresholds": {
356 + "mode": "absolute",
357 + "steps": [
358 + {
359 + "color": "green",
360 + "value": null
361 + },
362 + {
363 + "color": "red",
364 + "value": 80
365 + }
366 + ]
367 + },
368 + "unit": "s"
369 + },
370 + "overrides": []
371 + },
372 + "gridPos": {
373 + "h": 4,
374 + "w": 6,
375 + "x": 18,
376 + "y": 5
377 + },
378 + "id": 3,
379 + "options": {
380 + "colorMode": "value",
381 + "graphMode": "area",
382 + "justifyMode": "auto",
383 + "orientation": "auto",
384 + "percentChangeColorMode": "standard",
385 + "reduceOptions": {
386 + "calcs": [
387 + "lastNotNull"
388 + ],
389 + "fields": "",
390 + "values": false
391 + },
392 + "showPercentChange": false,
393 + "textMode": "auto",
394 + "wideLayout": true
395 + },
396 + "pluginVersion": "11.3.0",
397 + "targets": [
398 + {
399 + "editorMode": "code",
400 + "expr": "histogram_quantile(0.50, sum by(le) (rate(vllm:e2e_request_latency_seconds_bucket[$__range])))",
401 + "legendFormat": "__auto",
402 + "range": true,
403 + "refId": "A"
404 + }
405 + ],
406 + "title": "E2E Latency (P50)",
407 + "type": "stat"
408 + },
409 + {
410 + "collapsed": false,
411 + "gridPos": {
412 + "h": 1,
413 + "w": 24,
414 + "x": 0,
415 + "y": 9
416 + },
417 + "id": 8,
418 + "panels": [],
419 + "title": "Graph: TTFT(Time To First Token) over time ",
420 + "type": "row"
421 + },
422 + {
423 + "datasource": {
424 + "type": "prometheus",
425 + "uid": "${DS_PROMETHEUS}"
426 + },
427 + "description": "Time to first token (TTFT) latency, showing average and key percentiles over time.",
428 + "fieldConfig": {
429 + "defaults": {
430 + "color": {
431 + "mode": "palette-classic"
432 + },
433 + "custom": {
434 + "axisBorderShow": false,
435 + "axisCenteredZero": false,
436 + "axisColorMode": "text",
437 + "axisLabel": "Latency",
438 + "axisPlacement": "auto",
439 + "barAlignment": 0,
440 + "barWidthFactor": 0.6,
441 + "drawStyle": "line",
442 + "fillOpacity": 18,
443 + "gradientMode": "none",
444 + "hideFrom": {
445 + "legend": false,
446 + "tooltip": false,
447 + "viz": false
448 + },
449 + "insertNulls": false,
450 + "lineInterpolation": "linear",
451 + "lineWidth": 1,
452 + "pointSize": 5,
453 + "scaleDistribution": {
454 + "type": "linear"
455 + },
456 + "showPoints": "auto",
457 + "spanNulls": false,
458 + "stacking": {
459 + "group": "A",
460 + "mode": "none"
461 + },
462 + "thresholdsStyle": {
463 + "mode": "off"
464 + }
465 + },
466 + "decimals": 2,
467 + "mappings": [],
468 + "thresholds": {
469 + "mode": "absolute",
470 + "steps": [
471 + {
472 + "color": "green",
473 + "value": null
474 + },
475 + {
476 + "color": "red",
477 + "value": 80
478 + }
479 + ]
480 + },
481 + "unit": "s"
482 + },
483 + "overrides": []
484 + },
485 + "gridPos": {
486 + "h": 8,
487 + "w": 12,
488 + "x": 0,
489 + "y": 10
490 + },
491 + "id": 10,
492 + "options": {
493 + "legend": {
494 + "calcs": [],
495 + "displayMode": "list",
496 + "placement": "bottom",
497 + "showLegend": true
498 + },
499 + "tooltip": {
500 + "mode": "single",
501 + "sort": "none"
502 + }
503 + },
504 + "pluginVersion": "11.3.0",
505 + "targets": [
506 + {
507 + "editorMode": "code",
508 + "expr": "rate(vllm:time_to_first_token_seconds_sum[$__interval]) / rate(vllm:time_to_first_token_seconds_count[$__interval])",
509 + "format": "table",
510 + "legendFormat": "TTFT (Avg)",
511 + "range": true,
512 + "refId": "A"
513 + }
514 + ],
515 + "title": "TTFT Over Time",
516 + "type": "timeseries"
517 + },
518 + {
519 + "datasource": {
520 + "type": "prometheus",
521 + "uid": "${DS_PROMETHEUS}"
522 + },
523 + "description": "99th percentile of Time To First Token latency over the selected time range.",
524 + "fieldConfig": {
525 + "defaults": {
526 + "color": {
527 + "mode": "thresholds"
528 + },
529 + "decimals": 2,
530 + "displayName": "P99",
531 + "mappings": [],
532 + "thresholds": {
533 + "mode": "absolute",
534 + "steps": [
535 + {
536 + "color": "green",
537 + "value": null
538 + },
539 + {
540 + "color": "red",
541 + "value": 80
542 + }
543 + ]
544 + },
545 + "unit": "s"
546 + },
547 + "overrides": []
548 + },
549 + "gridPos": {
550 + "h": 4,
551 + "w": 6,
552 + "x": 12,
553 + "y": 10
554 + },
555 + "id": 14,
556 + "options": {
557 + "colorMode": "value",
558 + "graphMode": "area",
559 + "justifyMode": "auto",
560 + "orientation": "auto",
561 + "percentChangeColorMode": "standard",
562 + "reduceOptions": {
563 + "calcs": [
564 + "lastNotNull"
565 + ],
566 + "fields": "",
567 + "values": false
568 + },
569 + "showPercentChange": false,
570 + "textMode": "auto",
571 + "wideLayout": true
572 + },
573 + "pluginVersion": "11.3.0",
574 + "targets": [
575 + {
576 + "editorMode": "code",
577 + "expr": "histogram_quantile(0.99, sum by(le) (rate(vllm:time_to_first_token_seconds_bucket[$__range])))",
578 + "legendFormat": "TTFT (p99)",
579 + "range": true,
580 + "refId": "A"
581 + }
582 + ],
583 + "title": "TTFT (P99)",
584 + "type": "stat"
585 + },
586 + {
587 + "datasource": {
588 + "type": "prometheus",
589 + "uid": "${DS_PROMETHEUS}"
590 + },
591 + "description": "90th percentile of Time To First Token latency over the selected time range.",
592 + "fieldConfig": {
593 + "defaults": {
594 + "color": {
595 + "mode": "thresholds"
596 + },
597 + "decimals": 2,
598 + "displayName": "P90",
599 + "mappings": [],
600 + "thresholds": {
601 + "mode": "absolute",
602 + "steps": [
603 + {
604 + "color": "green",
605 + "value": null
606 + },
607 + {
608 + "color": "red",
609 + "value": 80
610 + }
611 + ]
612 + },
613 + "unit": "s"
614 + },
615 + "overrides": []
616 + },
617 + "gridPos": {
618 + "h": 4,
619 + "w": 6,
620 + "x": 18,
621 + "y": 10
622 + },
623 + "id": 13,
624 + "options": {
625 + "colorMode": "value",
626 + "graphMode": "area",
627 + "justifyMode": "auto",
628 + "orientation": "auto",
629 + "percentChangeColorMode": "standard",
630 + "reduceOptions": {
631 + "calcs": [
632 + "lastNotNull"
633 + ],
634 + "fields": "",
635 + "values": false
636 + },
637 + "showPercentChange": false,
638 + "textMode": "auto",
639 + "wideLayout": true
640 + },
641 + "pluginVersion": "11.3.0",
642 + "targets": [
643 + {
644 + "editorMode": "code",
645 + "expr": "histogram_quantile(0.90, sum by(le) (rate(vllm:time_to_first_token_seconds_bucket[$__range])))",
646 + "legendFormat": "TTFT (p90)",
647 + "range": true,
648 + "refId": "A"
649 + }
650 + ],
651 + "title": "TTFT (P90)",
652 + "type": "stat"
653 + },
654 + {
655 + "datasource": {
656 + "type": "prometheus",
657 + "uid": "${DS_PROMETHEUS}"
658 + },
659 + "description": "Average Time To First Token latency over the selected time range.",
660 + "fieldConfig": {
661 + "defaults": {
662 + "color": {
663 + "mode": "thresholds"
664 + },
665 + "decimals": 2,
666 + "displayName": "Average",
667 + "mappings": [],
668 + "thresholds": {
669 + "mode": "absolute",
670 + "steps": [
671 + {
672 + "color": "green",
673 + "value": null
674 + },
675 + {
676 + "color": "red",
677 + "value": 80
678 + }
679 + ]
680 + },
681 + "unit": "s"
682 + },
683 + "overrides": []
684 + },
685 + "gridPos": {
686 + "h": 4,
687 + "w": 6,
688 + "x": 12,
689 + "y": 14
690 + },
691 + "id": 11,
692 + "options": {
693 + "colorMode": "value",
694 + "graphMode": "area",
695 + "justifyMode": "auto",
696 + "orientation": "auto",
697 + "percentChangeColorMode": "standard",
698 + "reduceOptions": {
699 + "calcs": [
700 + "lastNotNull"
701 + ],
702 + "fields": "",
703 + "values": false
704 + },
705 + "showPercentChange": false,
706 + "textMode": "auto",
707 + "wideLayout": true
708 + },
709 + "pluginVersion": "11.3.0",
710 + "targets": [
711 + {
712 + "editorMode": "code",
713 + "expr": "(sum(increase(vllm:time_to_first_token_seconds_sum[$__range])) / sum(increase(vllm:time_to_first_token_seconds_count[$__range])))",
714 + "legendFormat": "__auto",
715 + "range": true,
716 + "refId": "A"
717 + }
718 + ],
719 + "title": "TTFT (Avg)",
720 + "type": "stat"
721 + },
722 + {
723 + "datasource": {
724 + "type": "prometheus",
725 + "uid": "${DS_PROMETHEUS}"
726 + },
727 + "description": "50th percentile (median) of Time To First Token latency over the selected time range.",
728 + "fieldConfig": {
729 + "defaults": {
730 + "color": {
731 + "mode": "thresholds"
732 + },
733 + "displayName": "P50",
734 + "mappings": [],
735 + "thresholds": {
736 + "mode": "absolute",
737 + "steps": [
738 + {
739 + "color": "green",
740 + "value": null
741 + },
742 + {
743 + "color": "red",
744 + "value": 80
745 + }
746 + ]
747 + },
748 + "unit": "s"
749 + },
750 + "overrides": []
751 + },
752 + "gridPos": {
753 + "h": 4,
754 + "w": 6,
755 + "x": 18,
756 + "y": 14
757 + },
758 + "id": 12,
759 + "options": {
760 + "colorMode": "value",
761 + "graphMode": "area",
762 + "justifyMode": "auto",
763 + "orietitletChangeColorMode": "standard",
764 + "reduceOptions": {
765 + "calcs": [
766 + "lastNotNull"
767 + ],
768 + "fields": "",
769 + "values": false
770 + },
771 + "showPercentChange": false,
772 + "textMode": "auto",
773 + "wideLayout": true
774 + },
775 + "pluginVersion": "11.3.0",
776 + "targets": [
777 + {
778 + "editorMode": "code",
779 + "expr": "histogram_quantile(0.50, sum by(le) (rate(vllm:time_to_first_token_seconds_bucket[$__range])))",
780 + "legendFormat": "__auto",
781 + "range": true,
782 + "refId": "A"
783 + }
784 + ],
785 + "title": "TTFT (P50)",
786 + "type": "stat"
787 + },
788 + {
789 + "collapsed": false,
790 + "gridPos": {
791 + "h": 1,
792 + "w": 24,
793 + "x": 0,
794 + "y": 18
795 + },
796 + "id": 7,
797 + "panels": [],
798 + "title": "ITL (Iteration Latency / Time Per Output Token) over time.",
799 + "type": "row"
800 + },
801 + {
802 + "datasource": {
803 + "type": "prometheus",
804 + "uid": "${DS_PROMETHEUS}"
805 + },
806 + "description": "Iteration latency, or average time taken to generate a single output token, with percentiles.",
807 + "fieldConfig": {
808 + "defaults": {
809 + "color": {
810 + "mode": "palette-classic"
811 + },
812 + "custom": {
813 + "axisBorderShow": false,
814 + "axisCenteredZero": false,
815 + "axisColorMode": "text",
816 + "axisLabel": "Latency",
817 + "axisPlacement": "auto",
818 + "barAlignment": 0,
819 + "barWidthFactor": 0.6,
820 + "drawStyle": "line",
821 + "fillOpacity": 17,
822 + "gradientMode": "none",
823 + "hideFrom": {
824 + "legend": false,
825 + "tooltip": false,
826 + "viz": false
827 + },
828 + "insertNulls": false,
829 + "lineInterpolation": "linear",
830 + "lineWidth": 1,
831 + "pointSize": 5,
832 + "scaleDistribution": {
833 + "type": "linear"
834 + },
835 + "showPoints": "auto",
836 + "spanNulls": false,
837 + "stacking": {
838 + "group": "A",
839 + "mode": "none"
840 + },
841 + "thresholdsStyle": {
842 + "mode": "off"
843 + }
844 + },
845 + "decimals": 2,
846 + "mappings": [],
847 + "thresholds": {
848 + "mode": "absolute",
849 + "steps": [
850 + {
851 + "color": "green",
852 + "value": null
853 + },
854 + {
855 + "color": "red",
856 + "value": 80
857 + }
858 + ]
859 + },
860 + "unit": "s"
861 + },
862 + "overrides": []
863 + },
864 + "gridPos": {
865 + "h": 8,
866 + "w": 12,
867 + "x": 0,
868 + "y": 19
869 + },
870 + "id": 15,
871 + "options": {
872 + "legend": {
873 + "calcs": [],
874 + "displayMode": "list",
875 + "placement": "bottom",
876 + "showLegend": true
877 + },
878 + "tooltip": {
879 + "mode": "single",
880 + "sort": "none"
881 + }
882 + },
883 + "pluginVersion": "11.3.0",
884 + "targets": [
885 + {
886 + "editorMode": "code",
887 + "expr": "rate(vllm:time_per_output_token_seconds_sum[$__interval]) / rate(vllm:time_per_output_token_seconds_count[$__interval])",
888 + "legendFormat": "ITL (Avg)",
889 + "range": true,
890 + "refId": "A"
891 + },
892 + {
893 + "datasource": {
894 + "type": "prometheus",
895 + "uid": "${DS_PROMETHEUS}"
896 + },
897 + "editorMode": "code",
898 + "expr": "histogram_quantile(0.50, sum by(le) (rate(vllm:time_per_output_token_seconds_bucket[$__interval])))",
899 + "hide": false,
900 + "instant": false,
901 + "legendFormat": "ITL (p50)",
902 + "range": true,
903 + "refId": "B"
904 + },
905 + {
906 + "datasource": {
907 + "type": "prometheus",
908 + "uid": "${DS_PROMETHEUS}"
909 + },
910 + "editorMode": "code",
911 + "expr": "histogram_quantile(0.90, sum by(le) (rate(vllm:time_per_output_token_seconds_bucket[$__interval])))",
912 + "hide": false,
913 + "instant": false,
914 + "legendFormat": "ITL (p90)",
915 + "range": true,
916 + "refId": "C"
917 + },
918 + {
919 + "datasource": {
920 + "type": "prometheus",
921 + "uid": "${DS_PROMETHEUS}"
922 + },
923 + "editorMode": "code",
924 + "expr": "histogram_quantile(0.99, sum by(le) (rate(vllm:time_per_output_token_seconds_bucket[$__interval])))",
925 + "hide": false,
926 + "instant": false,
927 + "legendFormat": "ITL (p99)",
928 + "range": true,
929 + "refId": "D"
930 + }
931 + ],
932 + "title": "ITL (Time Per Output Token) Over Time",
933 + "type": "timeseries"
934 + },
935 + {
936 + "datasource": {
937 + "type": "prometheus",
938 + "uid": "${DS_PROMETHEUS}"
939 + },
940 + "description": "90th percentile of Iteration Latency over the selected time range.",
941 + "fieldConfig": {
942 + "defaults": {
943 + "color": {
944 + "mode": "thresholds"
945 + },
946 + "decimals": 2,
947 + "mappings": [],
948 + "thresholds": {
949 + "mode": "absolute",
950 + "steps": [
951 + {
952 + "color": "green",
953 + "value": null
954 + },
955 + {
956 + "color": "red",
957 + "value": 80
958 + }
959 + ]
960 + },
961 + "unit": "s"
962 + },
963 + "overrides": []
964 + },
965 + "gridPos": {
966 + "h": 4,
967 + "w": 6,
968 + "x": 12,
969 + "y": 19
970 + },
971 + "id": 18,
972 + "options": {
973 + "colorMode": "value",
974 + "graphMode": "area",
975 + "justifyMode": "auto",
976 + "orientation": "auto",
977 + "percentChangeColorMode": "standard",
978 + "reduceOptions": {
979 + "calcs": [
980 + "lastNotNull"
981 + ],
982 + "fields": "",
983 + "values": false
984 + },
985 + "showPercentChange": false,
986 + "textMode": "auto",
987 + "wideLayout": true
988 + },
989 + "pluginVersion": "11.3.0",
990 + "targets": [
991 + {
992 + "editorMode": "code",
993 + "expr": "histogram_quantile(0.90, sum by(le) (rate(vllm:time_per_output_token_seconds_bucket[$__range])))",
994 + "legendFormat": "__auto",
995 + "range": true,
996 + "refId": "A"
997 + }
998 + ],
999 + "title": "ITL (P90)",
1000 + "type": "stat"
1001 + },
1002 + {
1003 + "datasource": {
1004 + "type": "prometheus",
1005 + "uid": "${DS_PROMETHEUS}"
1006 + },
1007 + "description": "99th percentile of Iteration Latency over the selected time range.\n\n",
1008 + "fieldConfig": {
1009 + "defaults": {
1010 + "color": {
1011 + "mode": "thresholds"
1012 + },
1013 + "decimals": 2,
1014 + "mappings": [],
1015 + "thresholds": {
1016 + "mode": "absolute",
1017 + "steps": [
1018 + {
1019 + "color": "green",
1020 + "value": null
1021 + },
1022 + {
1023 + "color": "red",
1024 + "value": 80
1025 + }
1026 + ]
1027 + },
1028 + "unit": "s"
1029 + },
1030 + "overrides": []
1031 + },
1032 + "gridPos": {
1033 + "h": 4,
1034 + "w": 6,
1035 + "x": 18,
1036 + "y": 19
1037 + },
1038 + "id": 19,
1039 + "options": {
1040 + "colorMode": "value",
1041 + "graphMode": "area",
1042 + "justifyMode": "auto",
1043 + "orientation": "auto",
1044 + "percentChangeColorMode": "standard",
1045 + "reduceOptions": {
1046 + "calcs": [
1047 + "lastNotNull"
1048 + ],
1049 + "fields": "",
1050 + "values": false
1051 + },
1052 + "showPercentChange": false,
1053 + "textMode": "auto",
1054 + "wideLayout": true
1055 + },
1056 + "pluginVersion": "11.3.0",
1057 + "targets": [
1058 + {
1059 + "editorMode": "code",
1060 + "expr": "histogram_quantile(0.99, sum by(le) (rate(vllm:time_per_output_token_seconds_bucket[$__range])))",
1061 + "legendFormat": "__auto",
1062 + "range": true,
1063 + "refId": "A"
1064 + }
1065 + ],
1066 + "title": "ITL (P99)",
1067 + "type": "stat"
1068 + },
1069 + {
1070 + "datasource": {
1071 + "type": "prometheus",
1072 + "uid": "${DS_PROMETHEUS}"
1073 + },
1074 + "description": "Average Iteration Latency (time per output token) over the selected time range.",
1075 + "fieldConfig": {
1076 + "defaults": {
1077 + "color": {
1078 + "mode": "thresholds"
1079 + },
1080 + "decimals": 2,
1081 + "mappings": [],
1082 + "thresholds": {
1083 + "mode": "absolute",
1084 + "steps": [
1085 + {
1086 + "color": "green",
1087 + "value": null
1088 + },
1089 + {
1090 + "color": "red",
1091 + "value": 80
1092 + }
1093 + ]
1094 + },
1095 + "unit": "s"
1096 + },
1097 + "overrides": []
1098 + },
1099 + "gridPos": {
1100 + "h": 4,
1101 + "w": 6,
1102 + "x": 12,
1103 + "y": 23
1104 + },
1105 + "id": 16,
1106 + "options": {
1107 + "colorMode": "value",
1108 + "graphMode": "area",
1109 + "justifyMode": "auto",
1110 + "orientation": "auto",
1111 + "percentChangeColorMode": "standard",
1112 + "reduceOptions": {
1113 + "calcs": [
1114 + "lastNotNull"
1115 + ],
1116 + "fields": "",
1117 + "values": false
1118 + },
1119 + "showPercentChange": false,
1120 + "textMode": "auto",
1121 + "wideLayout": true
1122 + },
1123 + "pluginVersion": "11.3.0",
1124 + "targets": [
1125 + {
1126 + "editorMode": "code",
1127 + "expr": "(sum(increase(vllm:time_per_output_token_seconds_sum[$__range])) / sum(increase(vllm:time_per_output_token_seconds_count[$__range])))",
1128 + "legendFormat": "__auto",
1129 + "range": true,
1130 + "refId": "A"
1131 + }
1132 + ],
1133 + "title": "ITL (Avg)",
1134 + "type": "stat"
1135 + },
1136 + {
1137 + "datasource": {
1138 + "type": "prometheus",
1139 + "uid": "${DS_PROMETHEUS}"
1140 + },
1141 + "description": "50th percentile (median) of Iteration Latency over the selected time range.",
1142 + "fieldConfig": {
1143 + "defaults": {
1144 + "color": {
1145 + "mode": "thresholds"
1146 + },
1147 + "decimals": 2,
1148 + "mappings": [],
1149 + "thresholds": {
1150 + "mode": "absolute",
1151 + "steps": [
1152 + {
1153 + "color": "green",
1154 + "value": null
1155 + },
1156 + {
1157 + "color": "red",
1158 + "value": 80
1159 + }
1160 + ]
1161 + },
1162 + "unit": "s"
1163 + },
1164 + "overrides": []
1165 + },
1166 + "gridPos": {
1167 + "h": 4,
1168 + "w": 6,
1169 + "x": 18,
1170 + "y": 23
1171 + },
1172 + "id": 17,
1173 + "options": {
1174 + "colorMode": "value",
1175 + "graphMode": "area",
1176 + "justifyMode": "auto",
1177 + "orientation": "auto",
1178 + "percentChangeColorMode": "standard",
1179 + "reduceOptions": {
1180 + "calcs": [
1181 + "lastNotNull"
1182 + ],
1183 + "fields": "",
1184 + "values": false
1185 + },
1186 + "showPercentChange": false,
1187 + "textMode": "auto",
1188 + "wideLayout": true
1189 + },
1190 + "pluginVersion": "11.3.0",
1191 + "targets": [
1192 + {
1193 + "editorMode": "code",
1194 + "expr": "histogram_quantile(0.50, sum by(le) (rate(vllm:time_per_output_token_seconds_bucket[$__range])))",
1195 + "legendFormat": "__auto",
1196 + "range": true,
1197 + "refId": "A"
1198 + }
1199 + ],
1200 + "title": "ITL (P50)",
1201 + "type": "stat"
1202 + },
1203 + {
1204 + "collapsed": false,
1205 + "gridPos": {
1206 + "h": 1,
1207 + "w": 24,
1208 + "x": 0,
1209 + "y": 27
1210 + },
1211 + "id": 6,
1212 + "panels": [],
1213 + "title": "TPS (Tokens Per Second)",
1214 + "type": "row"
1215 + },
1216 + {
1217 + "datasource": {
1218 + "type": "prometheus",
1219 + "uid": "${DS_PROMETHEUS}"
1220 + },
1221 + "description": "Rate of tokens processed per second, including prompt and generation phases.",
1222 + "fieldConfig": {
1223 + "defaults": {
1224 + "color": {
1225 + "mode": "palette-classic"
1226 + },
1227 + "custom": {
1228 + "axisBorderShow": false,
1229 + "axisCenteredZero": false,
1230 + "axisColorMode": "text",
1231 + "axisLabel": "",
1232 + "axisPlacement": "auto",
1233 + "barAlignment": 0,
1234 + "barWidthFactor": 0.6,
1235 + "drawStyle": "line",
1236 + "fillOpacity": 0,
1237 + "gradientMode": "none",
1238 + "hideFrom": {
1239 + "legend": false,
1240 + "tooltip": false,
1241 + "viz": false
1242 + },
1243 + "insertNulls": false,
1244 + "lineInterpolation": "linear",
1245 + "lineWidth": 1,
1246 + "pointSize": 5,
1247 + "scaleDistribution": {
1248 + "type": "linear"
1249 + },
1250 + "showPoints": "auto",
1251 + "spanNulls": false,
1252 + "stacking": {
1253 + "group": "A",
1254 + "mode": "none"
1255 + },
1256 + "thresholdsStyle": {
1257 + "mode": "off"
1258 + }
1259 + },
1260 + "mappings": [],
1261 + "thresholds": {
1262 + "mode": "absolute",
1263 + "steps": [
1264 + {
1265 + "color": "green",
1266 + "value": null
1267 + },
1268 + {
1269 + "color": "red",
1270 + "value": 80
1271 + }
1272 + ]
1273 + },
1274 + "unit": "tokens/sec (tps)"
1275 + },
1276 + "overrides": []
1277 + },
1278 + "gridPos": {
1279 + "h": 8,
1280 + "w": 12,
1281 + "x": 0,
1282 + "y": 28
1283 + },
1284 + "id": 20,
1285 + "options": {
1286 + "legend": {
1287 + "calcs": [],
1288 + "displayMode": "list",
1289 + "placement": "bottom",
1290 + "showLegend": true
1291 + },
1292 + "tooltip": {
1293 + "mode": "single",
1294 + "sort": "none"
1295 + }
1296 + },
1297 + "pluginVersion": "11.3.0",
1298 + "targets": [
1299 + {
1300 + "editorMode": "code",
1301 + "expr": "rate(vllm:generation_tokens_total[$__interval])",
1302 + "legendFormat": "Generation TPS",
1303 + "range": true,
1304 + "refId": "A"
1305 + },
1306 + {
1307 + "datasource": {
1308 + "type": "prometheus",
1309 + "uid": "${DS_PROMETHEUS}"
1310 + },
1311 + "editorMode": "code",
1312 + "expr": "rate(vllm:prompt_tokens_total[$__interval])",
1313 + "hide": false,
1314 + "instant": false,
1315 + "legendFormat": "Prompt TPS",
1316 + "range": true,
1317 + "refId": "B"
1318 + },
1319 + {
1320 + "datasource": {
1321 + "type": "prometheus",
1322 + "uid": "${DS_PROMETHEUS}"
1323 + },
1324 + "editorMode": "code",
1325 + "expr": "rate(vllm:iteration_tokens_total_count[$__interval])",
1326 + "hide": false,
1327 + "instant": false,
1328 + "legendFormat": "Overall Iteration TPS",
1329 + "range": true,
1330 + "refId": "C"
1331 + }
1332 + ],
1333 + "title": "TPS (Tokens Per Second) Over Time",
1334 + "type": "timeseries"
1335 + }
1336 + ],
1337 + "preload": false,
1338 + "schemaVersion": 40,
1339 + "tags": [],
1340 + "templating": {
1341 + "list": [
1342 + {
1343 + "name": "DS_PROMETHEUS",
1344 + "type": "datasource",
1345 + "label": "datasource",
1346 + "query": "prometheus",
1347 + "refresh": 1,
1348 + "current": {
1349 + "text": "Prometheus",
1350 + "value": "prometheus"
1351 + }
1352 + },
1353 + {
1354 + "current": {
1355 + "text": "avg : Average\n0.50 : P50\n0.90 : P90\n0.99 : P99\n0.999 : Max (Approx)",
1356 + "value": "avg : Average\n0.50 : P50\n0.90 : P90\n0.99 : P99\n0.999 : Max (Approx)"
1357 + },
1358 + "label": "Aggregation",
1359 + "name": "agg_method",
1360 + "options": [
1361 + {
1362 + "selected": true,
1363 + "text": "avg : Average\n0.50 : P50\n0.90 : P90\n0.99 : P99\n0.999 : Max (Approx)",
1364 + "value": "avg : Average\n0.50 : P50\n0.90 : P90\n0.99 : P99\n0.999 : Max (Approx)"
1365 + }
1366 + ],
1367 + "query": "avg : Average\n0.50 : P50\n0.90 : P90\n0.99 : P99\n0.999 : Max (Approx)",
1368 + "type": "custom"
1369 + },
1370 + {
1371 + "current": {
1372 + "text": [
1373 + "granite-33-2b-instruct"
1374 + ],
1375 + "value": [
1376 + "granite-33-2b-instruct"
1377 + ]
1378 + },
1379 + "definition": "label_values(vllm:generation_tokens_total,model_name)",
1380 + "includeAll": true,
1381 + "label": "Deployment_ID",
1382 + "multi": true,
1383 + "name": "Deployment_id",
1384 + "options": [],
1385 + "query": {
1386 + "qryType": 1,
1387 + "query": "label_values(vllm:generation_tokens_total,model_name)",
1388 + "refId": "PrometheusVariableQueryEditor-VariableQuery"
1389 + },
1390 + "refresh": 1,
1391 + "regex": "",
1392 + "type": "query"
1393 + }
1394 + ]
1395 + },
1396 + "time": {
1397 + "from": "now-12h",
1398 + "to": "now"
1399 + },
1400 + "timezone": "browser",
1401 + "uid": "performance-statistics",
1402 + "title": "Performance Statistics",
1403 + "version": 40,
1404 + "weekStart": ""
1405 +}
\ No newline at end of file
examples/online_serving/dashboards/grafana/query_statistics.json new
+760
@@ -0,0 +1,760 @@
1 +{
2 + "annotations": {
3 + "list": [
4 + {
5 + "builtIn": 1,
6 + "datasource": {
7 + "type": "grafana",
8 + "uid": "-- Grafana --"
9 + },
10 + "enable": true,
11 + "hide": true,
12 + "iconColor": "rgba(0, 211, 255, 1)",
13 + "name": "Annotations & Alerts",
14 + "type": "dashboard"
15 + }
16 + ]
17 + },
18 + "description": "High-level overview of VLLM model deployment behavior and key performance indicators. Designed for Data Scientists and Product Managers to monitor request volume, token throughput, and latency",
19 + "editable": true,
20 + "fiscalYearStartMonth": 0,
21 + "graphTooltip": 0,
22 + "id": 47,
23 + "links": [],
24 + "panels": [
25 + {
26 + "collapsed": true,
27 + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
28 + "id": 20,
29 + "panels": [],
30 + "title": "Request Over Time",
31 + "type": "row"
32 + },
33 + {
34 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
35 + "fieldConfig": {
36 + "defaults": {
37 + "color": { "mode": "palette-classic" },
38 + "custom": {
39 + "axisBorderShow": false,
40 + "axisCenteredZero": false,
41 + "axisColorMode": "text",
42 + "axisLabel": "",
43 + "axisPlacement": "auto",
44 + "barAlignment": 0,
45 + "barWidthFactor": 0.6,
46 + "drawStyle": "line",
47 + "fillOpacity": 0,
48 + "gradientMode": "none",
49 + "hideFrom": { "legend": false, "tooltip": false, "viz": false },
50 + "insertNulls": false,
51 + "lineInterpolation": "linear",
52 + "lineWidth": 1,
53 + "pointSize": 5,
54 + "scaleDistribution": { "type": "linear" },
55 + "showPoints": "auto",
56 + "spanNulls": false,
57 + "stacking": { "group": "A", "mode": "none" },
58 + "thresholdsStyle": { "mode": "off" }
59 + },
60 + "mappings": [],
61 + "thresholds": {
62 + "mode": "absolute",
63 + "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }]
64 + },
65 + "unit": "req/s"
66 + },
67 + "overrides": []
68 + },
69 + "gridPos": { "h": 6, "w": 10, "x": 0, "y": 1 },
70 + "id": 1,
71 + "options": {
72 + "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true },
73 + "tooltip": { "mode": "single", "sort": "none" }
74 + },
75 + "pluginVersion": "11.3.0",
76 + "targets": [
77 + {
78 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
79 + "editorMode": "code",
80 + "expr": "sum by (model_name) (\n rate(vllm:request_success_total{model_name=~\"$Deployment_id\"}[$__rate_interval])\n)",
81 + "interval": "1",
82 + "legendFormat": "{{model_name}}",
83 + "range": true,
84 + "refId": "A"
85 + }
86 + ],
87 + "title": "Successful Requests Over Time",
88 + "type": "timeseries"
89 + },
90 + {
91 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
92 + "fieldConfig": {
93 + "defaults": {
94 + "color": { "mode": "thresholds" },
95 + "mappings": [],
96 + "thresholds": {
97 + "mode": "absolute",
98 + "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }]
99 + },
100 + "unit": "req/s"
101 + },
102 + "overrides": []
103 + },
104 + "gridPos": { "h": 3, "w": 7, "x": 10, "y": 1 },
105 + "id": 2,
106 + "options": {
107 + "colorMode": "value",
108 + "graphMode": "area",
109 + "justifyMode": "auto",
110 + "orientation": "auto",
111 + "percentChangeColorMode": "standard",
112 + "reduceOptions": { "calcs": ["mean"], "fields": "", "values": false },
113 + "showPercentChange": false,
114 + "textMode": "auto",
115 + "wideLayout": true
116 + },
117 + "pluginVersion": "11.3.0",
118 + "targets": [
119 + {
120 + "editorMode": "code",
121 + "expr": "sum(rate(vllm:request_success_total{model_name=~\"$Deployment_id\"}[$__rate_interval]))",
122 + "legendFormat": "__auto",
123 + "range": true,
124 + "refId": "A"
125 + }
126 + ],
127 + "title": "Requests Avg Rate",
128 + "type": "stat"
129 + },
130 + {
131 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
132 + "fieldConfig": {
133 + "defaults": {
134 + "color": { "mode": "thresholds" },
135 + "mappings": [
136 + { "options": { "Calcultaions": { "index": 0, "text": "Last (not null)" } }, "type": "value" }
137 + ],
138 + "thresholds": {
139 + "mode": "absolute",
140 + "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }]
141 + },
142 + "unit": "ms"
143 + },
144 + "overrides": []
145 + },
146 + "gridPos": { "h": 3, "w": 7, "x": 17, "y": 1 },
147 + "id": 3,
148 + "options": {
149 + "colorMode": "value",
150 + "graphMode": "area",
151 + "justifyMode": "auto",
152 + "orientation": "auto",
153 + "percentChangeColorMode": "standard",
154 + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
155 + "showPercentChange": false,
156 + "textMode": "auto",
157 + "wideLayout": true
158 + },
159 + "pluginVersion": "11.3.0",
160 + "targets": [
161 + {
162 + "editorMode": "code",
163 + "expr": "histogram_quantile(0.50, sum by(le, model_name) (rate(vllm:e2e_request_latency_seconds_bucket{model_name=~\"$Deployment_id\"}[$__rate_interval])))",
164 + "legendFormat": "__auto",
165 + "range": true,
166 + "refId": "A"
167 + }
168 + ],
169 + "title": "p50 Latency",
170 + "type": "stat"
171 + },
172 + {
173 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
174 + "fieldConfig": {
175 + "defaults": {
176 + "color": { "mode": "thresholds" },
177 + "mappings": [
178 + { "options": { "Calculation": { "index": 0, "text": "Last (not null)" } }, "type": "value" }
179 + ],
180 + "thresholds": {
181 + "mode": "absolute",
182 + "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }]
183 + },
184 + "unit": "ms"
185 + },
186 + "overrides": []
187 + },
188 + "gridPos": { "h": 3, "w": 7, "x": 10, "y": 4 },
189 + "id": 4,
190 + "options": {
191 + "colorMode": "value",
192 + "graphMode": "area",
193 + "justifyMode": "auto",
194 + "orientation": "auto",
195 + "percentChangeColorMode": "standard",
196 + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
197 + "showPercentChange": false,
198 + "textMode": "auto",
199 + "wideLayout": true
200 + },
201 + "pluginVersion": "11.3.0",
202 + "targets": [
203 + {
204 + "editorMode": "code",
205 + "expr": "histogram_quantile(0.90, sum by(le, model_name) (rate(vllm:e2e_request_latency_seconds_bucket{model_name=~\"$Deployment_id\"}[$__rate_interval])))",
206 + "legendFormat": "__auto",
207 + "range": true,
208 + "refId": "A"
209 + }
210 + ],
211 + "title": "p90 Latency",
212 + "type": "stat"
213 + },
214 + {
215 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
216 + "fieldConfig": {
217 + "defaults": {
218 + "color": { "mode": "thresholds" },
219 + "mappings": [
220 + { "options": { "Calculation": { "index": 0, "text": "Last (not null)" } }, "type": "value" }
221 + ],
222 + "thresholds": {
223 + "mode": "absolute",
224 + "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }]
225 + },
226 + "unit": "ms"
227 + },
228 + "overrides": []
229 + },
230 + "gridPos": { "h": 3, "w": 7, "x": 17, "y": 4 },
231 + "id": 5,
232 + "options": {
233 + "colorMode": "value",
234 + "graphMode": "area",
235 + "justifyMode": "auto",
236 + "orientation": "auto",
237 + "percentChangeColorMode": "standard",
238 + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
239 + "showPercentChange": false,
240 + "textMode": "auto",
241 + "wideLayout": true
242 + },
243 + "pluginVersion": "11.3.0",
244 + "targets": [
245 + {
246 + "editorMode": "code",
247 + "expr": "histogram_quantile(0.99, sum by(le, model_name) (rate(vllm:e2e_request_latency_seconds_bucket{model_name=~\"$Deployment_id\"}[$__rate_interval])))",
248 + "legendFormat": "__auto",
249 + "range": true,
250 + "refId": "A"
251 + }
252 + ],
253 + "title": "p99 Latency",
254 + "type": "stat"
255 + },
256 + {
257 + "collapsed": false,
258 + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 7 },
259 + "id": 19,
260 + "panels": [],
261 + "title": "Size Distribution",
262 + "type": "row"
263 + },
264 + {
265 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
266 + "fieldConfig": {
267 + "defaults": {
268 + "color": { "mode": "palette-classic" },
269 + "custom": {
270 + "fillOpacity": 80,
271 + "gradientMode": "none",
272 + "hideFrom": { "legend": false, "tooltip": false, "viz": false },
273 + "lineWidth": 1,
274 + "stacking": { "group": "A", "mode": "none" }
275 + },
276 + "mappings": [],
277 + "thresholds": {
278 + "mode": "absolute",
279 + "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }]
280 + },
281 + "unit": "cps"
282 + },
283 + "overrides": []
284 + },
285 + "gridPos": { "h": 6, "w": 10, "x": 0, "y": 8 },
286 + "id": 6,
287 + "options": {
288 + "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true },
289 + "tooltip": { "mode": "single", "sort": "none" }
290 + },
291 + "pluginVersion": "11.3.0",
292 + "targets": [
293 + {
294 + "editorMode": "code",
295 + "expr": "sum by (le, model_name) (rate(vllm:request_prompt_tokens_bucket{model_name=~\"$Deployment_id\"}[$__rate_interval]))",
296 + "legendFormat": "{{model_name}} le={{le}}",
297 + "range": true,
298 + "refId": "A"
299 + }
300 + ],
301 + "title": "Input Token Size Distribution",
302 + "type": "histogram"
303 + },
304 + {
305 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
306 + "fieldConfig": {
307 + "defaults": {
308 + "color": { "mode": "thresholds" },
309 + "mappings": [
310 + { "options": { "calculation ": { "index": 0, "text": "Last (not null)" } }, "type": "value" }
311 + ],
312 + "thresholds": {
313 + "mode": "absolute",
314 + "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }]
315 + },
316 + "unit": "cps"
317 + },
318 + "overrides": []
319 + },
320 + "gridPos": { "h": 3, "w": 7, "x": 10, "y": 8 },
321 + "id": 9,
322 + "options": {
323 + "colorMode": "value",
324 + "graphMode": "area",
325 + "justifyMode": "auto",
326 + "orientation": "auto",
327 + "percentChangeColorMode": "standard",
328 + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
329 + "showPercentChange": false,
330 + "textMode": "auto",
331 + "wideLayout": true
332 + },
333 + "pluginVersion": "11.3.0",
334 + "targets": [
335 + {
336 + "editorMode": "code",
337 + "expr": "histogram_quantile(0.90, sum by(le, model_name) (rate(vllm:request_prompt_tokens_bucket{model_name=~\"$Deployment_id\"}[$__rate_interval])))",
338 + "legendFormat": "__auto",
339 + "range": true,
340 + "refId": "A"
341 + }
342 + ],
343 + "title": "Input Token Size p90",
344 + "type": "stat"
345 + },
346 + {
347 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
348 + "fieldConfig": {
349 + "defaults": {
350 + "color": { "mode": "thresholds" },
351 + "mappings": [
352 + { "options": { "Calcultion": { "index": 0, "text": "Last (not null)" } }, "type": "value" }
353 + ],
354 + "thresholds": {
355 + "mode": "absolute",
356 + "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }]
357 + },
358 + "unit": "cps"
359 + },
360 + "overrides": []
361 + },
362 + "gridPos": { "h": 3, "w": 7, "x": 17, "y": 8 },
363 + "id": 8,
364 + "options": {
365 + "colorMode": "value",
366 + "graphMode": "area",
367 + "justifyMode": "auto",
368 + "orientation": "auto",
369 + "percentChangeColorMode": "standard",
370 + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
371 + "showPercentChange": false,
372 + "textMode": "auto",
373 + "wideLayout": true
374 + },
375 + "pluginVersion": "11.3.0",
376 + "targets": [
377 + {
378 + "editorMode": "code",
379 + "expr": "histogram_quantile(0.50, sum by(le, model_name) (rate(vllm:request_prompt_tokens_bucket{model_name=~\"$Deployment_id\"}[$__rate_interval])))",
380 + "legendFormat": "__auto",
381 + "range": true,
382 + "refId": "A"
383 + }
384 + ],
385 + "title": "Input Token Size p50",
386 + "type": "stat"
387 + },
388 + {
389 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
390 + "fieldConfig": {
391 + "defaults": {
392 + "color": { "mode": "thresholds" },
393 + "mappings": [
394 + { "options": { "Calcultaion": { "index": 0, "text": "mean" } }, "type": "value" }
395 + ],
396 + "thresholds": {
397 + "mode": "absolute",
398 + "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }]
399 + },
400 + "unit": "cps"
401 + },
402 + "overrides": []
403 + },
404 + "gridPos": { "h": 3, "w": 7, "x": 10, "y": 11 },
405 + "id": 7,
406 + "options": {
407 + "colorMode": "value",
408 + "graphMode": "area",
409 + "justifyMode": "auto",
410 + "orientation": "auto",
411 + "percentChangeColorMode": "standard",
412 + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
413 + "showPercentChange": false,
414 + "textMode": "auto",
415 + "wideLayout": true
416 + },
417 + "pluginVersion": "11.3.0",
418 + "targets": [
419 + {
420 + "editorMode": "code",
421 + "expr": "sum(rate(vllm:prompt_tokens_total{model_name=~\"$Deployment_id\"}[$__rate_interval]))\n/\nsum(rate(vllm:request_success_total{model_name=~\"$Deployment_id\"}[$__rate_interval]))",
422 + "legendFormat": "__auto",
423 + "range": true,
424 + "refId": "A"
425 + }
426 + ],
427 + "title": "Input Token Size Avg",
428 + "type": "stat"
429 + },
430 + {
431 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
432 + "fieldConfig": {
433 + "defaults": {
434 + "color": { "mode": "thresholds" },
435 + "mappings": [
436 + { "options": { "Calculation": { "index": 0, "text": "Last (not null)" } }, "type": "value" }
437 + ],
438 + "thresholds": {
439 + "mode": "absolute",
440 + "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }]
441 + },
442 + "unit": "cps"
443 + },
444 + "overrides": []
445 + },
446 + "gridPos": { "h": 3, "w": 7, "x": 17, "y": 11 },
447 + "id": 10,
448 + "options": {
449 + "colorMode": "value",
450 + "graphMode": "area",
451 + "justifyMode": "auto",
452 + "orientation": "auto",
453 + "percentChangeColorMode": "standard",
454 + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
455 + "showPercentChange": false,
456 + "textMode": "auto",
457 + "wideLayout": true
458 + },
459 + "pluginVersion": "11.3.0",
460 + "targets": [
461 + {
462 + "editorMode": "code",
463 + "expr": "histogram_quantile(0.99, sum by(le, model_name) (rate(vllm:request_prompt_tokens_bucket{model_name=~\"$Deployment_id\"}[$__rate_interval])))",
464 + "legendFormat": "__auto",
465 + "range": true,
466 + "refId": "A"
467 + }
468 + ],
469 + "title": "Input Token Size p99",
470 + "type": "stat"
471 + },
472 + {
473 + "collapsed": true,
474 + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 14 },
475 + "id": 18,
476 + "panels": [],
477 + "title": "Input Token Over Time",
478 + "type": "row"
479 + },
480 + {
481 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
482 + "fieldConfig": {
483 + "defaults": {
484 + "color": { "mode": "palette-classic" },
485 + "custom": {
486 + "axisBorderShow": false,
487 + "axisCenteredZero": false,
488 + "axisColorMode": "text",
489 + "axisLabel": "",
490 + "axisPlacement": "auto",
491 + "barAlignment": 0,
492 + "barWidthFactor": 0.6,
493 + "drawStyle": "line",
494 + "fillOpacity": 0,
495 + "gradientMode": "none",
496 + "hideFrom": { "legend": false, "tooltip": false, "viz": false },
497 + "insertNulls": false,
498 + "lineInterpolation": "linear",
499 + "lineWidth": 1,
500 + "pointSize": 5,
501 + "scaleDistribution": { "type": "linear" },
502 + "showPoints": "auto",
503 + "spanNulls": false,
504 + "stacking": { "group": "A", "mode": "none" },
505 + "thresholdsStyle": { "mode": "off" }
506 + },
507 + "mappings": [],
508 + "thresholds": {
509 + "mode": "absolute",
510 + "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }]
511 + },
512 + "unit": "cps"
513 + },
514 + "overrides": []
515 + },
516 + "gridPos": { "h": 6, "w": 10, "x": 0, "y": 15 },
517 + "id": 11,
518 + "options": {
519 + "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true },
520 + "tooltip": { "mode": "single", "sort": "none" }
521 + },
522 + "pluginVersion": "11.3.0",
523 + "targets": [
524 + {
525 + "editorMode": "code",
526 + "expr": "sum by (model_name) (rate(vllm:prompt_tokens_total{model_name=~\"$Deployment_id\"}[$__rate_interval]))",
527 + "legendFormat": "{{model_name}}",
528 + "range": true,
529 + "refId": "A"
530 + }
531 + ],
532 + "title": "Input Tokens Over Time",
533 + "type": "timeseries"
534 + },
535 + {
536 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
537 + "fieldConfig": {
538 + "defaults": {
539 + "color": { "mode": "thresholds" },
540 + "mappings": [
541 + { "options": { "Calculation": { "index": 0, "text": "mean" } }, "type": "value" }
542 + ],
543 + "thresholds": {
544 + "mode": "absolute",
545 + "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }]
546 + },
547 + "unit": "cps"
548 + },
549 + "overrides": []
550 + },
551 + "gridPos": { "h": 3, "w": 7, "x": 10, "y": 15 },
552 + "id": 12,
553 + "options": {
554 + "colorMode": "value",
555 + "graphMode": "area",
556 + "justifyMode": "auto",
557 + "orientation": "auto",
558 + "percentChangeColorMode": "standard",
559 + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
560 + "showPercentChange": false,
561 + "textMode": "auto",
562 + "wideLayout": true
563 + },
564 + "pluginVersion": "11.3.0",
565 + "targets": [
566 + {
567 + "editorMode": "code",
568 + "expr": "sum(rate(vllm:prompt_tokens_total{model_name=~\"$Deployment_id\"}[$__rate_interval]))",
569 + "legendFormat": "__auto",
570 + "range": true,
571 + "refId": "A"
572 + }
573 + ],
574 + "title": "Input Tokens/Sec Avg",
575 + "type": "stat"
576 + },
577 + {
578 + "collapsed": false,
579 + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 21 },
580 + "id": 17,
581 + "panels": [],
582 + "title": "Output Token Over Time",
583 + "type": "row"
584 + },
585 + {
586 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
587 + "fieldConfig": {
588 + "defaults": {
589 + "color": { "mode": "palette-classic" },
590 + "custom": {
591 + "axisBorderShow": false,
592 + "axisCenteredZero": false,
593 + "axisColorMode": "text",
594 + "axisLabel": "",
595 + "axisPlacement": "auto",
596 + "barAlignment": 0,
597 + "barWidthFactor": 0.6,
598 + "drawStyle": "line",
599 + "fillOpacity": 0,
600 + "gradientMode": "none",
601 + "hideFrom": { "legend": false, "tooltip": false, "viz": false },
602 + "insertNulls": false,
603 + "lineInterpolation": "linear",
604 + "lineWidth": 1,
605 + "pointSize": 5,
606 + "scaleDistribution": { "type": "linear" },
607 + "showPoints": "auto",
608 + "spanNulls": false,
609 + "stacking": { "group": "A", "mode": "none" },
610 + "thresholdsStyle": { "mode": "off" }
611 + },
612 + "mappings": [],
613 + "thresholds": {
614 + "mode": "absolute",
615 + "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }]
616 + },
617 + "unit": "cps"
618 + },
619 + "overrides": []
620 + },
621 + "gridPos": { "h": 6, "w": 10, "x": 0, "y": 22 },
622 + "id": 13,
623 + "options": {
624 + "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true },
625 + "tooltip": { "mode": "single", "sort": "none" }
626 + },
627 + "pluginVersion": "11.3.0",
628 + "targets": [
629 + {
630 + "editorMode": "code",
631 + "expr": "sum by (model_name) (rate(vllm:generation_tokens_total{model_name=~\"$Deployment_id\"}[$__rate_interval]))",
632 + "legendFormat": "{{model_name}}",
633 + "range": true,
634 + "refId": "A"
635 + }
636 + ],
637 + "title": "Output Tokens Over Time",
638 + "type": "timeseries"
639 + },
640 + {
641 + "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
642 + "fieldConfig": {
643 + "defaults": {
644 + "color": { "mode": "thresholds" },
645 + "mappings": [
646 + { "options": { "Calculation": { "index": 0, "text": "mean" } }, "type": "value" }
647 + ],
648 + "thresholds": {
649 + "mode": "absolute",
650 + "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }]
651 + },
652 + "unit": "cps"
653 + },
654 + "overrides": []
655 + },
656 + "gridPos": { "h": 3, "w": 7, "x": 10, "y": 22 },
657 + "id": 14,
658 + "options": {
659 + "colorMode": "value",
660 + "graphMode": "area",
661 + "justifyMode": "auto",
662 + "orientation": "auto",
663 + "percentChangeColorMode": "standard",
664 + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
665 + "showPercentChange": false,
666 + "textMode": "auto",
667 + "wideLayout": true
668 + },
669 + "pluginVersion": "11.3.0",
670 + "targets": [
671 + {
672 + "editorMode": "code",
673 + "expr": "sum(rate(vllm:generation_tokens_total{model_name=~\"$Deployment_id\"}[$__rate_interval]))",
674 + "legendFormat": "__auto",
675 + "range": true,
676 + "refId": "A"
677 + }
678 + ],
679 + "title": "Output Tokens/Sec Avg",
680 + "type": "stat"
681 + }
682 + ],
683 + "preload": false,
684 + "schemaVersion": 40,
685 + "tags": [],
686 + "templating": {
687 + "list": [
688 + {
689 + "current": { "text": "Prometheus", "value": "4184fc20-68a7-483a-8d9b-7caa59c680dd" },
690 + "label": "datasource",
691 + "name": "DS_PROMETHEUS",
692 + "options": [],
693 + "query": "prometheus",
694 + "refresh": 1,
695 + "type": "datasource"
696 + },
697 + {
698 + "current": { "text": ["All"], "value": ["$__all"] },
699 + "definition": "label_values(vllm:request_success_total,model_name)",
700 + "includeAll": true,
701 + "label": "Deployment_ID",
702 + "multi": true,
703 + "name": "Deployment_id",
704 + "options": [],
705 + "query": {
706 + "qryType": 1,
707 + "query": "label_values(vllm:request_success_total,model_name)",
708 + "refId": "PrometheusVariableQueryEditor-VariableQuery"
709 + },
710 + "refresh": 1,
711 + "regex": "",
712 + "sort": 1,
713 + "type": "query"
714 + },
715 + {
716 + "current": { "text": "All hours", "value": "All hours" },
717 + "hide": 2,
718 + "label": "Rush Hours Only",
719 + "name": "rush_hours",
720 + "options": [
721 + { "selected": true, "text": "false", "value": "All hours" },
722 + { "selected": false, "text": "true", "value": "Rush hours" }
723 + ],
724 + "query": "false : All hours, true : Rush hours",
725 + "type": "custom"
726 + },
727 + {
728 + "current": { "text": "All", "value": "All" },
729 + "hide": 2,
730 + "label": "Rush Hours Type",
731 + "name": "rush_hours_type",
732 + "options": [
733 + { "selected": true, "text": "^All__.*$", "value": "All" },
734 + { "selected": false, "text": "^Static__.*$", "value": "Static" },
735 + { "selected": false, "text": "^Dynamic__.*$", "value": "Dynamic" }
736 + ],
737 + "query": "^All__.*$ : All, ^Static__.*$ : Static, ^Dynamic__.*$ : Dynamic",
738 + "type": "custom"
739 + },
740 + {
741 + "current": { "text": "", "value": "" },
742 + "hide": 2,
743 + "name": "query0",
744 + "options": [],
745 + "query": "",
746 + "refresh": 1,
747 + "regex": "",
748 + "type": "query"
749 + }
750 + ]
751 + },
752 + "time": { "from": "now-12h", "to": "now" },
753 + "timepicker": {},
754 + "timezone": "browser",
755 + "title": "Query Statistics_New4",
756 + "uid": "query-statistics4",
757 + "version": 2,
758 + "weekStart": ""
759 +}
760 +
examples/online_serving/dashboards/perses/README.md new
+48
@@ -0,0 +1,48 @@
1 +# Perses Dashboards for vLLM Monitoring
2 +
3 +This directory contains Perses dashboard configurations designed to monitor vLLM
4 +performance and metrics.
5 +
6 +## Requirements
7 +
8 +- Perses instance (standalone or via operator)
9 +- Prometheus data source configured in Perses
10 +- vLLM deployment with Prometheus metrics enabled
11 +
12 +## Dashboard Format
13 +
14 +We provide dashboards in the **native Perses YAML format** that works across all
15 +deployment methods:
16 +
17 +- **Files**: `*.yaml` (native Perses dashboard specifications)
18 +- **Format**: Pure dashboard specifications that work everywhere
19 +- **Usage**: Works with standalone Perses, API imports, CLI, and file provisioning
20 +- **Kubernetes**: Directly compatible with Perses Operator
21 +
22 +## Dashboard Descriptions
23 +
24 +- **[performance_statistics.yaml](./performance_statistics.yaml)**: Performance metrics with aggregated latency
25 + statistics
26 +- **[query_statistics.yaml](./query_statistics.yaml)**: Query performance and deployment metrics
27 +
28 +## Deployment Options
29 +
30 +### Direct Import to Perses
31 +
32 +Import the dashboard specifications via Perses API or CLI:
33 +
34 +```bash
35 +percli apply -f performance_statistics.yaml
36 +```
37 +
38 +### Perses Operator (Kubernetes)
39 +
40 +The native YAML format works directly with the Perses Operator:
41 +
42 +```bash
43 +kubectl apply -f performance_statistics.yaml -n <namespace>
44 +```
45 +
46 +### File Provisioning
47 +
48 +Place the YAML files in a Perses provisioning folder for automatic loading.
examples/online_serving/dashboards/perses/performance_statistics.yaml new
+764
@@ -0,0 +1,764 @@
1 +kind: PersesDashboard
2 +metadata:
3 + name: performance-statistics
4 + createdAt: 0001-01-01T00:00:00Z
5 + updatedAt: 0001-01-01T00:00:00Z
6 + version: 0
7 + project: ""
8 +spec:
9 + display:
10 + name: Performance Statistics
11 +
12 + variables:
13 + - kind: ListVariable
14 + spec:
15 + display:
16 + name: Deployment_ID
17 + hidden: false
18 + name: Deployment_id
19 + allowAllValue: true
20 + allowMultiple: true
21 + defaultValue:
22 + - $__all
23 + sort: alphabetical-asc
24 + plugin:
25 + kind: PrometheusLabelValuesVariable
26 + spec:
27 + datasource:
28 + kind: PrometheusDatasource
29 + name: accelerators-thanos-querier-datasource
30 + labelName: model_name
31 + matchers:
32 + # Any one vllm metric that always carries model_name
33 + - vllm:generation_tokens_total{}
34 +
35 + panels:
36 + "1":
37 + kind: Panel
38 + spec:
39 + display:
40 + name: E2E Latency over Time
41 + plugin:
42 + kind: TimeSeriesChart
43 + spec:
44 + legend:
45 + mode: table
46 + position: bottom
47 + queries:
48 + - kind: TimeSeriesQuery
49 + spec:
50 + plugin:
51 + kind: PrometheusTimeSeriesQuery
52 + spec:
53 + datasource:
54 + kind: PrometheusDatasource
55 + name: accelerators-thanos-querier-datasource
56 + # avg latency by model = sum(rate(sum)) / sum(rate(count))
57 + query: >
58 + sum by (model_name) (rate(vllm:e2e_request_latency_seconds_sum{model_name=~"$Deployment_id"}[$__interval]))
59 + /
60 + sum by (model_name) (rate(vllm:e2e_request_latency_seconds_count{model_name=~"$Deployment_id"}[$__interval]))
61 + seriesNameFormat: '{{model_name}}'
62 +
63 + "2":
64 + kind: Panel
65 + spec:
66 + display:
67 + name: E2E Latency (Avg)
68 + plugin:
69 + kind: StatChart
70 + spec:
71 + calculation: last-number
72 + queries:
73 + - kind: TimeSeriesQuery
74 + spec:
75 + plugin:
76 + kind: PrometheusTimeSeriesQuery
77 + spec:
78 + datasource:
79 + kind: PrometheusDatasource
80 + name: accelerators-thanos-querier-datasource
81 + query: >
82 + (sum by (model_name) (increase(vllm:e2e_request_latency_seconds_sum{model_name=~"$Deployment_id"}[$__range])))
83 + /
84 + (sum by (model_name) (increase(vllm:e2e_request_latency_seconds_count{model_name=~"$Deployment_id"}[$__range])))
85 +
86 + "3":
87 + kind: Panel
88 + spec:
89 + display:
90 + name: E2E Latency (P50)
91 + plugin:
92 + kind: StatChart
93 + spec:
94 + calculation: last-number
95 + queries:
96 + - kind: TimeSeriesQuery
97 + spec:
98 + plugin:
99 + kind: PrometheusTimeSeriesQuery
100 + spec:
101 + datasource:
102 + kind: PrometheusDatasource
103 + name: accelerators-thanos-querier-datasource
104 + query: >
105 + histogram_quantile(
106 + 0.50,
107 + sum by (le, model_name) (
108 + rate(vllm:e2e_request_latency_seconds_bucket{model_name=~"$Deployment_id"}[$__interval])
109 + )
110 + )
111 +
112 + "4":
113 + kind: Panel
114 + spec:
115 + display:
116 + name: E2E Latency (P90)
117 + plugin:
118 + kind: StatChart
119 + spec:
120 + calculation: last-number
121 + queries:
122 + - kind: TimeSeriesQuery
123 + spec:
124 + plugin:
125 + kind: PrometheusTimeSeriesQuery
126 + spec:
127 + datasource:
128 + kind: PrometheusDatasource
129 + name: accelerators-thanos-querier-datasource
130 + query: >
131 + histogram_quantile(
132 + 0.90,
133 + sum by (le, model_name) (
134 + rate(vllm:e2e_request_latency_seconds_bucket{model_name=~"$Deployment_id"}[$__interval])
135 + )
136 + )
137 +
138 + "5":
139 + kind: Panel
140 + spec:
141 + display:
142 + name: E2E Latency (P99)
143 + plugin:
144 + kind: StatChart
145 + spec:
146 + calculation: last-number
147 + queries:
148 + - kind: TimeSeriesQuery
149 + spec:
150 + plugin:
151 + kind: PrometheusTimeSeriesQuery
152 + spec:
153 + datasource:
154 + kind: PrometheusDatasource
155 + name: accelerators-thanos-querier-datasource
156 + query: >
157 + histogram_quantile(
158 + 0.99,
159 + sum by (le, model_name) (
160 + rate(vllm:e2e_request_latency_seconds_bucket{model_name=~"$Deployment_id"}[$__interval])
161 + )
162 + )
163 +
164 + "6":
165 + kind: Panel
166 + spec:
167 + display:
168 + name: TTFT over Time
169 + plugin:
170 + kind: TimeSeriesChart
171 + spec:
172 + legend:
173 + mode: table
174 + position: bottom
175 + queries:
176 + - kind: TimeSeriesQuery
177 + spec:
178 + plugin:
179 + kind: PrometheusTimeSeriesQuery
180 + spec:
181 + datasource:
182 + kind: PrometheusDatasource
183 + name: accelerators-thanos-querier-datasource
184 + query: >
185 + sum by (model_name) (rate(vllm:time_to_first_token_seconds_sum{model_name=~"$Deployment_id"}[$__interval]))
186 + /
187 + sum by (model_name) (rate(vllm:time_to_first_token_seconds_count{model_name=~"$Deployment_id"}[$__interval]))
188 + seriesNameFormat: '{{model_name}}'
189 +
190 + "7":
191 + kind: Panel
192 + spec:
193 + display:
194 + name: TTFT (Avg)
195 + plugin:
196 + kind: StatChart
197 + spec:
198 + calculation: last-number
199 + queries:
200 + - kind: TimeSeriesQuery
201 + spec:
202 + plugin:
203 + kind: PrometheusTimeSeriesQuery
204 + spec:
205 + datasource:
206 + kind: PrometheusDatasource
207 + name: accelerators-thanos-querier-datasource
208 + query: >
209 + (sum by (model_name) (increase(vllm:time_to_first_token_seconds_sum{model_name=~"$Deployment_id"}[$__range])))
210 + /
211 + (sum by (model_name) (increase(vllm:time_to_first_token_seconds_count{model_name=~"$Deployment_id"}[$__range])))
212 +
213 + "8":
214 + kind: Panel
215 + spec:
216 + display:
217 + name: TTFT (P50)
218 + plugin:
219 + kind: StatChart
220 + spec:
221 + calculation: last-number
222 + queries:
223 + - kind: TimeSeriesQuery
224 + spec:
225 + plugin:
226 + kind: PrometheusTimeSeriesQuery
227 + spec:
228 + datasource:
229 + kind: PrometheusDatasource
230 + name: accelerators-thanos-querier-datasource
231 + query: >
232 + histogram_quantile(
233 + 0.50,
234 + sum by (le, model_name) (
235 + rate(vllm:time_to_first_token_seconds_bucket{model_name=~"$Deployment_id"}[$__interval])
236 + )
237 + )
238 +
239 + "9":
240 + kind: Panel
241 + spec:
242 + display:
243 + name: TTFT (P90)
244 + plugin:
245 + kind: StatChart
246 + spec:
247 + calculation: last-number
248 + queries:
249 + - kind: TimeSeriesQuery
250 + spec:
251 + plugin:
252 + kind: PrometheusTimeSeriesQuery
253 + spec:
254 + datasource:
255 + kind: PrometheusDatasource
256 + name: accelerators-thanos-querier-datasource
257 + query: >
258 + histogram_quantile(
259 + 0.90,
260 + sum by (le, model_name) (
261 + rate(vllm:time_to_first_token_seconds_bucket{model_name=~"$Deployment_id"}[$__interval])
262 + )
263 + )
264 +
265 + "10":
266 + kind: Panel
267 + spec:
268 + display:
269 + name: TTFT (P99)
270 + plugin:
271 + kind: StatChart
272 + spec:
273 + calculation: last-number
274 + queries:
275 + - kind: TimeSeriesQuery
276 + spec:
277 + plugin:
278 + kind: PrometheusTimeSeriesQuery
279 + spec:
280 + datasource:
281 + kind: PrometheusDatasource
282 + name: accelerators-thanos-querier-datasource
283 + query: >
284 + histogram_quantile(
285 + 0.99,
286 + sum by (le, model_name) (
287 + rate(vllm:time_to_first_token_seconds_bucket{model_name=~"$Deployment_id"}[$__interval])
288 + )
289 + )
290 +
291 + "11":
292 + kind: Panel
293 + spec:
294 + display:
295 + name: ITL (Time per Output Token) over Time
296 + plugin:
297 + kind: TimeSeriesChart
298 + spec:
299 + legend:
300 + mode: table
301 + position: bottom
302 + queries:
303 + - kind: TimeSeriesQuery
304 + spec:
305 + plugin:
306 + kind: PrometheusTimeSeriesQuery
307 + spec:
308 + datasource:
309 + kind: PrometheusDatasource
310 + name: accelerators-thanos-querier-datasource
311 + query: >
312 + sum by (model_name) (rate(vllm:time_per_output_token_seconds_sum{model_name=~"$Deployment_id"}[$__interval]))
313 + /
314 + sum by (model_name) (rate(vllm:time_per_output_token_seconds_count{model_name=~"$Deployment_id"}[$__interval]))
315 + seriesNameFormat: '{{model_name}}'
316 + - kind: TimeSeriesQuery
317 + spec:
318 + plugin:
319 + kind: PrometheusTimeSeriesQuery
320 + spec:
321 + datasource:
322 + kind: PrometheusDatasource
323 + name: accelerators-thanos-querier-datasource
324 + query: >
325 + histogram_quantile(
326 + 0.50,
327 + sum by (le, model_name) (
328 + rate(vllm:time_per_output_token_seconds_bucket{model_name=~"$Deployment_id"}[$__interval])
329 + )
330 + )
331 + seriesNameFormat: '{{model_name}} p50'
332 + - kind: TimeSeriesQuery
333 + spec:
334 + plugin:
335 + kind: PrometheusTimeSeriesQuery
336 + spec:
337 + datasource:
338 + kind: PrometheusDatasource
339 + name: accelerators-thanos-querier-datasource
340 + query: >
341 + histogram_quantile(
342 + 0.90,
343 + sum by (le, model_name) (
344 + rate(vllm:time_per_output_token_seconds_bucket{model_name=~"$Deployment_id"}[$__interval])
345 + )
346 + )
347 + seriesNameFormat: '{{model_name}} p90'
348 + - kind: TimeSeriesQuery
349 + spec:
350 + plugin:
351 + kind: PrometheusTimeSeriesQuery
352 + spec:
353 + datasource:
354 + kind: PrometheusDatasource
355 + name: accelerators-thanos-querier-datasource
356 + query: >
357 + histogram_quantile(
358 + 0.99,
359 + sum by (le, model_name) (
360 + rate(vllm:time_per_output_token_seconds_bucket{model_name=~"$Deployment_id"}[$__interval])
361 + )
362 + )
363 + seriesNameFormat: '{{model_name}} p99'
364 +
365 + "12":
366 + kind: Panel
367 + spec:
368 + display:
369 + name: ITL (Avg)
370 + plugin:
371 + kind: StatChart
372 + spec:
373 + calculation: last-number
374 + queries:
375 + - kind: TimeSeriesQuery
376 + spec:
377 + plugin:
378 + kind: PrometheusTimeSeriesQuery
379 + spec:
380 + datasource:
381 + kind: PrometheusDatasource
382 + name: accelerators-thanos-querier-datasource
383 + query: >
384 + (sum by (model_name) (increase(vllm:time_per_output_token_seconds_sum{model_name=~"$Deployment_id"}[$__range])))
385 + /
386 + (sum by (model_name) (increase(vllm:time_per_output_token_seconds_count{model_name=~"$Deployment_id"}[$__range])))
387 +
388 + "13":
389 + kind: Panel
390 + spec:
391 + display:
392 + name: ITL (P50)
393 + plugin:
394 + kind: StatChart
395 + spec:
396 + calculation: last-number
397 + queries:
398 + - kind: TimeSeriesQuery
399 + spec:
400 + plugin:
401 + kind: PrometheusTimeSeriesQuery
402 + spec:
403 + datasource:
404 + kind: PrometheusDatasource
405 + name: accelerators-thanos-querier-datasource
406 + query: >
407 + histogram_quantile(
408 + 0.50,
409 + sum by (le, model_name) (
410 + rate(vllm:time_per_output_token_seconds_bucket{model_name=~"$Deployment_id"}[$__interval])
411 + )
412 + )
413 +
414 + "14":
415 + kind: Panel
416 + spec:
417 + display:
418 + name: ITL (P90)
419 + plugin:
420 + kind: StatChart
421 + spec:
422 + calculation: last-number
423 + queries:
424 + - kind: TimeSeriesQuery
425 + spec:
426 + plugin:
427 + kind: PrometheusTimeSeriesQuery
428 + spec:
429 + datasource:
430 + kind: PrometheusDatasource
431 + name: accelerators-thanos-querier-datasource
432 + query: >
433 + histogram_quantile(
434 + 0.90,
435 + sum by (le, model_name) (
436 + rate(vllm:time_per_output_token_seconds_bucket{model_name=~"$Deployment_id"}[$__interval])
437 + )
438 + )
439 +
440 + "15":
441 + kind: Panel
442 + spec:
443 + display:
444 + name: ITL (P99)
445 + plugin:
446 + kind: StatChart
447 + spec:
448 + calculation: last-number
449 + queries:
450 + - kind: TimeSeriesQuery
451 + spec:
452 + plugin:
453 + kind: PrometheusTimeSeriesQuery
454 + spec:
455 + datasource:
456 + kind: PrometheusDatasource
457 + name: accelerators-thanos-querier-datasource
458 + query: >
459 + histogram_quantile(
460 + 0.99,
461 + sum by (le, model_name) (
462 + rate(vllm:time_per_output_token_seconds_bucket{model_name=~"$Deployment_id"}[$__interval])
463 + )
464 + )
465 +
466 + "16":
467 + kind: Panel
468 + spec:
469 + display:
470 + name: TPS (Tokens/sec) over Time
471 + plugin:
472 + kind: TimeSeriesChart
473 + spec:
474 + legend:
475 + mode: table
476 + position: bottom
477 + queries:
478 + - kind: TimeSeriesQuery
479 + spec:
480 + plugin:
481 + kind: PrometheusTimeSeriesQuery
482 + spec:
483 + datasource:
484 + kind: PrometheusDatasource
485 + name: accelerators-thanos-querier-datasource
486 + query: >
487 + sum by (model_name) (rate(vllm:generation_tokens_total{model_name=~"$Deployment_id"}[$__interval]))
488 + seriesNameFormat: '{{model_name}} generation'
489 + - kind: TimeSeriesQuery
490 + spec:
491 + plugin:
492 + kind: PrometheusTimeSeriesQuery
493 + spec:
494 + datasource:
495 + kind: PrometheusDatasource
496 + name: accelerators-thanos-querier-datasource
497 + query: >
498 + sum by (model_name) (rate(vllm:prompt_tokens_total{model_name=~"$Deployment_id"}[$__interval]))
499 + seriesNameFormat: '{{model_name}} prompt'
500 + - kind: TimeSeriesQuery
501 + spec:
502 + plugin:
503 + kind: PrometheusTimeSeriesQuery
504 + spec:
505 + datasource:
506 + kind: PrometheusDatasource
507 + name: accelerators-thanos-querier-datasource
508 + # overall iteration tokens/sec if exposed
509 + query: >
510 + rate(vllm:iteration_tokens_total_count[$__interval])
511 + seriesNameFormat: 'iteration overall'
512 +
513 + "17":
514 + kind: Panel
515 + spec:
516 + display:
517 + name: KV Cache Usage (avg %)
518 + plugin:
519 + kind: StatChart
520 + spec:
521 + calculation: last-number
522 + queries:
523 + - kind: TimeSeriesQuery
524 + spec:
525 + plugin:
526 + kind: PrometheusTimeSeriesQuery
527 + spec:
528 + datasource:
529 + kind: PrometheusDatasource
530 + name: accelerators-thanos-querier-datasource
531 + # Multiply by 100 so we can read it as a percentage without setting a unit (avoids CUE unit conflicts)
532 + query: >
533 + 100 * avg(vllm:gpu_cache_usage_perc)
534 +
535 + "18":
536 + kind: Panel
537 + spec:
538 + display:
539 + name: Running Requests by Pod
540 + plugin:
541 + kind: TimeSeriesChart
542 + spec:
543 + legend:
544 + mode: table
545 + position: bottom
546 + queries:
547 + - kind: TimeSeriesQuery
548 + spec:
549 + plugin:
550 + kind: PrometheusTimeSeriesQuery
551 + spec:
552 + datasource:
553 + kind: PrometheusDatasource
554 + name: accelerators-thanos-querier-datasource
555 + query: >
556 + sum by (pod) (vllm:num_requests_running)
557 + seriesNameFormat: '{{pod}}'
558 +
559 + "19":
560 + kind: Panel
561 + spec:
562 + display:
563 + name: Waiting Requests by Pod
564 + plugin:
565 + kind: TimeSeriesChart
566 + spec:
567 + legend:
568 + mode: table
569 + position: bottom
570 + queries:
571 + - kind: TimeSeriesQuery
572 + spec:
573 + plugin:
574 + kind: PrometheusTimeSeriesQuery
575 + spec:
576 + datasource:
577 + kind: PrometheusDatasource
578 + name: accelerators-thanos-querier-datasource
579 + query: >
580 + sum by (pod) (vllm:num_requests_waiting)
581 + seriesNameFormat: '{{pod}}'
582 +
583 + "20":
584 + kind: Panel
585 + spec:
586 + display:
587 + name: Running Requests (sum)
588 + plugin:
589 + kind: StatChart
590 + spec:
591 + calculation: last-number
592 + queries:
593 + - kind: TimeSeriesQuery
594 + spec:
595 + plugin:
596 + kind: PrometheusTimeSeriesQuery
597 + spec:
598 + datasource:
599 + kind: PrometheusDatasource
600 + name: accelerators-thanos-querier-datasource
601 + query: sum(vllm:num_requests_running)
602 +
603 + "21":
604 + kind: Panel
605 + spec:
606 + display:
607 + name: Waiting Requests (sum)
608 + plugin:
609 + kind: StatChart
610 + spec:
611 + calculation: last-number
612 + queries:
613 + - kind: TimeSeriesQuery
614 + spec:
615 + plugin:
616 + kind: PrometheusTimeSeriesQuery
617 + spec:
618 + datasource:
619 + kind: PrometheusDatasource
620 + name: accelerators-thanos-querier-datasource
621 + query: sum(vllm:num_requests_waiting)
622 +
623 + layouts:
624 + - kind: Grid
625 + spec:
626 + display:
627 + title: Overview
628 + items:
629 + - x: 0
630 + y: 0
631 + width: 6
632 + height: 3
633 + content: { $ref: '#/spec/panels/17' } # KV cache %
634 + - x: 6
635 + y: 0
636 + width: 6
637 + height: 3
638 + content: { $ref: '#/spec/panels/20' } # running sum
639 + - x: 12
640 + y: 0
641 + width: 6
642 + height: 3
643 + content: { $ref: '#/spec/panels/21' } # waiting sum
644 +
645 + - kind: Grid
646 + spec:
647 + display:
648 + title: E2E Latency
649 + items:
650 + - x: 0
651 + y: 1
652 + width: 10
653 + height: 6
654 + content: { $ref: '#/spec/panels/1' }
655 + - x: 10
656 + y: 1
657 + width: 7
658 + height: 3
659 + content: { $ref: '#/spec/panels/2' }
660 + - x: 17
661 + y: 1
662 + width: 7
663 + height: 3
664 + content: { $ref: '#/spec/panels/3' }
665 + - x: 10
666 + y: 4
667 + width: 7
668 + height: 3
669 + content: { $ref: '#/spec/panels/4' }
670 + - x: 17
671 + y: 4
672 + width: 7
673 + height: 3
674 + content: { $ref: '#/spec/panels/5' }
675 +
676 + - kind: Grid
677 + spec:
678 + display:
679 + title: TTFT
680 + items:
681 + - x: 0
682 + y: 8
683 + width: 10
684 + height: 6
685 + content: { $ref: '#/spec/panels/6' }
686 + - x: 10
687 + y: 8
688 + width: 7
689 + height: 3
690 + content: { $ref: '#/spec/panels/7' }
691 + - x: 17
692 + y: 8
693 + width: 7
694 + height: 3
695 + content: { $ref: '#/spec/panels/8' }
696 + - x: 10
697 + y: 11
698 + width: 7
699 + height: 3
700 + content: { $ref: '#/spec/panels/9' }
701 + - x: 17
702 + y: 11
703 + width: 7
704 + height: 3
705 + content: { $ref: '#/spec/panels/10' }
706 +
707 + - kind: Grid
708 + spec:
709 + display:
710 + title: ITL (Time per Output Token)
711 + items:
712 + - x: 0
713 + y: 15
714 + width: 10
715 + height: 6
716 + content: { $ref: '#/spec/panels/11' }
717 + - x: 10
718 + y: 15
719 + width: 7
720 + height: 3
721 + content: { $ref: '#/spec/panels/12' }
722 + - x: 17
723 + y: 15
724 + width: 7
725 + height: 3
726 + content: { $ref: '#/spec/panels/13' }
727 + - x: 10
728 + y: 18
729 + width: 7
730 + height: 3
731 + content: { $ref: '#/spec/panels/14' }
732 + - x: 17
733 + y: 18
734 + width: 7
735 + height: 3
736 + content: { $ref: '#/spec/panels/15' }
737 +
738 + - kind: Grid
739 + spec:
740 + display:
741 + title: TPS (Prompt / Generation / Iteration)
742 + items:
743 + - x: 0
744 + y: 22
745 + width: 14
746 + height: 6
747 + content: { $ref: '#/spec/panels/16' }
748 +
749 + - kind: Grid
750 + spec:
751 + display:
752 + title: Per-Pod Request State
753 + items:
754 + - x: 0
755 + y: 28
756 + width: 12
757 + height: 6
758 + content: { $ref: '#/spec/panels/18' }
759 + - x: 12
760 + y: 28
761 + width: 12
762 + height: 6
763 + content: { $ref: '#/spec/panels/19' }
764 +
examples/online_serving/dashboards/perses/query_statistics.yaml new
+392
@@ -0,0 +1,392 @@
1 +kind: PersesDashboard
2 +metadata:
3 + name: query-statistics
4 + createdAt: 0001-01-01T00:00:00Z
5 + updatedAt: 0001-01-01T00:00:00Z
6 + version: 0
7 + project: ""
8 +spec:
9 + display:
10 + name: Query Statistics_New
11 +
12 + variables:
13 + - kind: ListVariable
14 + spec:
15 + name: NS
16 + display: { name: Namespace }
17 + allowMultiple: false
18 + defaultValue: llm-d
19 + plugin:
20 + kind: PrometheusLabelValuesVariable
21 + spec:
22 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
23 + labelName: namespace
24 + matchers:
25 + - up{service=~".*vllm.*"}
26 +
27 + - kind: ListVariable
28 + spec:
29 + name: SVC
30 + display: { name: Service }
31 + allowMultiple: false
32 + defaultValue: vllm-qwen2-0-5b-sim
33 + plugin:
34 + kind: PrometheusLabelValuesVariable
35 + spec:
36 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
37 + labelName: service
38 + matchers:
39 + - up{namespace="$NS",service=~".*vllm.*"}
40 +
41 + - kind: ListVariable
42 + spec:
43 + name: MODEL
44 + display: { name: Model (real vLLM) }
45 + allowAllValue: true
46 + allowMultiple: true
47 + defaultValue: ["$__all"]
48 + plugin:
49 + kind: PrometheusLabelValuesVariable
50 + spec:
51 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
52 + labelName: model_name
53 + matchers:
54 + - vllm:request_success_total{namespace="$NS",service="$SVC"}
55 +
56 + panels:
57 +
58 + # --- Core (works on Simulator & Real) ---
59 + core_running_now:
60 + kind: Panel
61 + spec:
62 + display: { name: Running Requests (now) }
63 + plugin: { kind: StatChart, spec: { calculation: last-number } }
64 + queries:
65 + - kind: TimeSeriesQuery
66 + spec:
67 + plugin:
68 + kind: PrometheusTimeSeriesQuery
69 + spec:
70 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
71 + query: sum(vllm:num_requests_running{namespace="$NS",service="$SVC"}) or vector(0)
72 + minStep: "15s"
73 +
74 + core_waiting_now:
75 + kind: Panel
76 + spec:
77 + display: { name: Waiting Requests (now) }
78 + plugin: { kind: StatChart, spec: { calculation: last-number } }
79 + queries:
80 + - kind: TimeSeriesQuery
81 + spec:
82 + plugin:
83 + kind: PrometheusTimeSeriesQuery
84 + spec:
85 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
86 + query: sum(vllm:num_requests_waiting{namespace="$NS",service="$SVC"}) or vector(0)
87 + minStep: "15s"
88 +
89 + core_kv_usage_now:
90 + kind: Panel
91 + spec:
92 + display: { name: KV Cache Usage (0–1) }
93 + plugin: { kind: StatChart, spec: { calculation: last-number } }
94 + queries:
95 + - kind: TimeSeriesQuery
96 + spec:
97 + plugin:
98 + kind: PrometheusTimeSeriesQuery
99 + spec:
100 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
101 + query: avg(vllm:gpu_cache_usage_perc{namespace="$NS",service="$SVC"}) or vector(0)
102 + minStep: "15s"
103 +
104 + core_running_ts:
105 + kind: Panel
106 + spec:
107 + display: { name: Running Over Time }
108 + plugin:
109 + kind: TimeSeriesChart
110 + spec:
111 + legend: { mode: table, position: bottom }
112 + visual: { display: line, lineWidth: 1, areaOpacity: 0.3 }
113 + queries:
114 + - kind: TimeSeriesQuery
115 + spec:
116 + plugin:
117 + kind: PrometheusTimeSeriesQuery
118 + spec:
119 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
120 + query: sum by (service) (vllm:num_requests_running{namespace="$NS",service="$SVC"}) or vector(0)
121 + minStep: "15s"
122 +
123 + core_waiting_ts:
124 + kind: Panel
125 + spec:
126 + display: { name: Waiting Over Time }
127 + plugin:
128 + kind: TimeSeriesChart
129 + spec:
130 + legend: { mode: table, position: bottom }
131 + visual: { display: line, lineWidth: 1, areaOpacity: 0.3 }
132 + queries:
133 + - kind: TimeSeriesQuery
134 + spec:
135 + plugin:
136 + kind: PrometheusTimeSeriesQuery
137 + spec:
138 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
139 + query: sum by (service) (vllm:num_requests_waiting{namespace="$NS",service="$SVC"}) or vector(0)
140 + minStep: "15s"
141 +
142 + core_targets_up:
143 + kind: Panel
144 + spec:
145 + display: { name: Scrape Targets Up }
146 + plugin: { kind: StatChart, spec: { calculation: last-number } }
147 + queries:
148 + - kind: TimeSeriesQuery
149 + spec:
150 + plugin:
151 + kind: PrometheusTimeSeriesQuery
152 + spec:
153 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
154 + query: count(up{namespace="$NS",service="$SVC"} == 1) or vector(0)
155 + minStep: "15s"
156 +
157 + # --- KV Cache as Percent (works on Simulator & Real) ---
158 + core_kv_usage_pct_now:
159 + kind: Panel
160 + spec:
161 + display: { name: KV Cache Usage (%) – now }
162 + plugin: { kind: StatChart, spec: { calculation: last-number } }
163 + queries:
164 + - kind: TimeSeriesQuery
165 + spec:
166 + plugin:
167 + kind: PrometheusTimeSeriesQuery
168 + spec:
169 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
170 + # multiply by 100 to present percentage; omit format.unit to avoid schema conflicts
171 + query: (avg(vllm:gpu_cache_usage_perc{namespace="$NS",service="$SVC"}) * 100) or vector(0)
172 + minStep: "15s"
173 +
174 + core_kv_usage_pct_ts:
175 + kind: Panel
176 + spec:
177 + display: { name: KV Cache Usage (%) – over time }
178 + plugin:
179 + kind: TimeSeriesChart
180 + spec:
181 + legend: { mode: table, position: bottom }
182 + visual: { display: line, lineWidth: 1, areaOpacity: 0.3 }
183 + queries:
184 + - kind: TimeSeriesQuery
185 + spec:
186 + plugin:
187 + kind: PrometheusTimeSeriesQuery
188 + spec:
189 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
190 + query: (avg by (service) (vllm:gpu_cache_usage_perc{namespace="$NS",service="$SVC"}) * 100) or vector(0)
191 + minStep: "15s"
192 +
193 + # --- Per-Pod breakdowns (works on Simulator & Real) ---
194 + per_pod_running_ts:
195 + kind: Panel
196 + spec:
197 + display: { name: Running by Pod }
198 + plugin:
199 + kind: TimeSeriesChart
200 + spec:
201 + legend: { mode: table, position: bottom }
202 + visual: { display: line, lineWidth: 1, areaOpacity: 0.3 }
203 + queries:
204 + - kind: TimeSeriesQuery
205 + spec:
206 + plugin:
207 + kind: PrometheusTimeSeriesQuery
208 + spec:
209 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
210 + query: sum by (pod) (vllm:num_requests_running{namespace="$NS",service="$SVC"}) or vector(0)
211 + minStep: "15s"
212 +
213 + per_pod_waiting_ts:
214 + kind: Panel
215 + spec:
216 + display: { name: Waiting by Pod }
217 + plugin:
218 + kind: TimeSeriesChart
219 + spec:
220 + legend: { mode: table, position: bottom }
221 + visual: { display: line, lineWidth: 1, areaOpacity: 0.3 }
222 + queries:
223 + - kind: TimeSeriesQuery
224 + spec:
225 + plugin:
226 + kind: PrometheusTimeSeriesQuery
227 + spec:
228 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
229 + query: sum by (pod) (vllm:num_requests_waiting{namespace="$NS",service="$SVC"}) or vector(0)
230 + minStep: "15s"
231 +
232 + per_pod_kv_pct_ts:
233 + kind: Panel
234 + spec:
235 + display: { name: KV Cache (%) by Pod }
236 + plugin:
237 + kind: TimeSeriesChart
238 + spec:
239 + legend: { mode: table, position: bottom }
240 + visual: { display: line, lineWidth: 1, areaOpacity: 0.3 }
241 + queries:
242 + - kind: TimeSeriesQuery
243 + spec:
244 + plugin:
245 + kind: PrometheusTimeSeriesQuery
246 + spec:
247 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
248 + # if your exporter labels kv metric with pod (the sim does), this works; otherwise it will just return empty
249 + query: (avg by (pod) (vllm:gpu_cache_usage_perc{namespace="$NS",service="$SVC"}) * 100) or vector(0)
250 + minStep: "15s"
251 +
252 + # --- Real vLLM only (zeros on simulator) ---
253 + real_req_rate_ts:
254 + kind: Panel
255 + spec:
256 + display: { name: Request Rate (real vLLM) }
257 + plugin:
258 + kind: TimeSeriesChart
259 + spec:
260 + legend: { mode: table, position: bottom }
261 + visual: { display: line, lineWidth: 1, areaOpacity: 0.3 }
262 + queries:
263 + - kind: TimeSeriesQuery
264 + spec:
265 + plugin:
266 + kind: PrometheusTimeSeriesQuery
267 + spec:
268 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
269 + query: sum by (model_name) (rate(vllm:request_success_total{namespace="$NS",service="$SVC",model_name=~"$MODEL"}[$__interval])) or vector(0)
270 + minStep: "15s"
271 +
272 + real_p50:
273 + kind: Panel
274 + spec:
275 + display: { name: p50 Latency (real vLLM) }
276 + plugin: { kind: StatChart, spec: { calculation: last-number } }
277 + queries:
278 + - kind: TimeSeriesQuery
279 + spec:
280 + plugin:
281 + kind: PrometheusTimeSeriesQuery
282 + spec:
283 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
284 + query: histogram_quantile(0.50, sum by (le, model_name) (rate(vllm:e2e_request_latency_seconds_bucket{namespace="$NS",service="$SVC",model_name=~"$MODEL"}[$__interval]))) or vector(0)
285 + minStep: "15s"
286 +
287 + real_p90:
288 + kind: Panel
289 + spec:
290 + display: { name: p90 Latency (real vLLM) }
291 + plugin: { kind: StatChart, spec: { calculation: last-number } }
292 + queries:
293 + - kind: TimeSeriesQuery
294 + spec:
295 + plugin:
296 + kind: PrometheusTimeSeriesQuery
297 + spec:
298 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
299 + query: histogram_quantile(0.90, sum by (le, model_name) (rate(vllm:e2e_request_latency_seconds_bucket{namespace="$NS",service="$SVC",model_name=~"$MODEL"}[$__interval]))) or vector(0)
300 + minStep: "15s"
301 +
302 + real_p99:
303 + kind: Panel
304 + spec:
305 + display: { name: p99 Latency (real vLLM) }
306 + plugin: { kind: StatChart, spec: { calculation: last-number } }
307 + queries:
308 + - kind: TimeSeriesQuery
309 + spec:
310 + plugin:
311 + kind: PrometheusTimeSeriesQuery
312 + spec:
313 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
314 + query: histogram_quantile(0.99, sum by (le, model_name) (rate(vllm:e2e_request_latency_seconds_bucket{namespace="$NS",service="$SVC",model_name=~"$MODEL"}[$__interval]))) or vector(0)
315 + minStep: "15s"
316 +
317 + real_input_tokens_ts:
318 + kind: Panel
319 + spec:
320 + display: { name: Input Tokens / sec (real vLLM) }
321 + plugin:
322 + kind: TimeSeriesChart
323 + spec:
324 + legend: { mode: table, position: bottom }
325 + visual: { display: line, lineWidth: 1, areaOpacity: 0.3 }
326 + queries:
327 + - kind: TimeSeriesQuery
328 + spec:
329 + plugin:
330 + kind: PrometheusTimeSeriesQuery
331 + spec:
332 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
333 + query: sum by (model_name) (rate(vllm:prompt_tokens_total{namespace="$NS",service="$SVC",model_name=~"$MODEL"}[$__interval])) or vector(0)
334 + minStep: "15s"
335 +
336 + real_output_tokens_ts:
337 + kind: Panel
338 + spec:
339 + display: { name: Output Tokens / sec (real vLLM) }
340 + plugin:
341 + kind: TimeSeriesChart
342 + spec:
343 + legend: { mode: table, position: bottom }
344 + visual: { display: line, lineWidth: 1, areaOpacity: 0.3 }
345 + queries:
346 + - kind: TimeSeriesQuery
347 + spec:
348 + plugin:
349 + kind: PrometheusTimeSeriesQuery
350 + spec:
351 + datasource: { kind: PrometheusDatasource, name: accelerators-thanos-querier-datasource }
352 + query: sum by (model_name) (rate(vllm:generation_tokens_total{namespace="$NS",service="$SVC",model_name=~"$MODEL"}[$__interval])) or vector(0)
353 + minStep: "15s"
354 +
355 + layouts:
356 + - kind: Grid
357 + spec:
358 + display: { title: Core (Sim & Real) }
359 + items:
360 + - { x: 0, y: 0, width: 6, height: 3, content: { $ref: '#/spec/panels/core_running_now' } }
361 + - { x: 6, y: 0, width: 6, height: 3, content: { $ref: '#/spec/panels/core_waiting_now' } }
362 + - { x: 12, y: 0, width: 6, height: 3, content: { $ref: '#/spec/panels/core_kv_usage_now' } }
363 + - { x: 18, y: 0, width: 6, height: 3, content: { $ref: '#/spec/panels/core_targets_up' } }
364 + - { x: 0, y: 3, width: 12, height: 6, content: { $ref: '#/spec/panels/core_running_ts' } }
365 + - { x: 12, y: 3, width: 12, height: 6, content: { $ref: '#/spec/panels/core_waiting_ts' } }
366 +
367 + - kind: Grid
368 + spec:
369 + display: { title: KV Cache (%) }
370 + items:
371 + - { x: 0, y: 9, width: 6, height: 3, content: { $ref: '#/spec/panels/core_kv_usage_pct_now' } }
372 + - { x: 6, y: 9, width: 18, height: 6, content: { $ref: '#/spec/panels/core_kv_usage_pct_ts' } }
373 +
374 + - kind: Grid
375 + spec:
376 + display: { title: Per-Pod breakdowns }
377 + items:
378 + - { x: 0, y: 15, width: 12, height: 6, content: { $ref: '#/spec/panels/per_pod_running_ts' } }
379 + - { x: 12, y: 15, width: 12, height: 6, content: { $ref: '#/spec/panels/per_pod_waiting_ts' } }
380 + - { x: 0, y: 21, width: 24, height: 6, content: { $ref: '#/spec/panels/per_pod_kv_pct_ts' } }
381 +
382 + - kind: Grid
383 + spec:
384 + display: { title: Real vLLM only (shows 0 on simulator) }
385 + items:
386 + - { x: 0, y: 27, width: 12, height: 6, content: { $ref: '#/spec/panels/real_req_rate_ts' } }
387 + - { x: 12, y: 27, width: 4, height: 3, content: { $ref: '#/spec/panels/real_p50' } }
388 + - { x: 16, y: 27, width: 4, height: 3, content: { $ref: '#/spec/panels/real_p90' } }
389 + - { x: 20, y: 27, width: 4, height: 3, content: { $ref: '#/spec/panels/real_p99' } }
390 + - { x: 0, y: 33, width: 12, height: 6, content: { $ref: '#/spec/panels/real_input_tokens_ts' } }
391 + - { x: 12, y: 33, width: 12, height: 6, content: { $ref: '#/spec/panels/real_output_tokens_ts' } }
392 +