A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 14:45 UTC
[V1][Metrics] Fix http metrics middleware (#15894)
Mark McLoughlin committed
Apr 17, 2025 at 20:52 UTC
e4755f7fac67e9dd1723173f2eb30778dbe8d792
2 files changed
+29
-18
docs/source/design/v1/metrics.md
+11
@@ -86,6 +86,17 @@ See [the PR which added this Dashboard](gh-pr:2316) for interesting and useful b
86
87
Prometheus support was initially added [using the aioprometheus library](gh-pr:1890), but a switch was made quickly to [prometheus_client](gh-pr:2730). The rationale is discussed in both linked PRs.
88
89
+With the switch to `aioprometheus`, we lost a `MetricsMiddleware` to track HTTP metrics, but this was reinstated [using prometheus_fastapi_instrumentator](gh-pr:15657):
90
+
91
+```bash
92
+$ curl http://0.0.0.0:8000/metrics 2>/dev/null | grep -P '^http_(?!.*(_bucket|_created|_sum)).*'
93
+http_requests_total{handler="/v1/completions",method="POST",status="2xx"} 201.0
94
+http_request_size_bytes_count{handler="/v1/completions"} 201.0
95
+http_response_size_bytes_count{handler="/v1/completions"} 201.0
96
+http_request_duration_highr_seconds_count 201.0
97
+http_request_duration_seconds_count{handler="/v1/completions",method="POST"} 201.0
98
+```
99
+
100
### Multi-process Mode
101
102
In v0, metrics are collected in the engine core process and we use multi-process mode to make them available in the API server process. See <gh-pr:7279>.
vllm/entrypoints/openai/api_server.py
+18
-18
@@ -310,33 +310,33 @@ def mount_metrics(app: FastAPI):
310
# We need to set PROMETHEUS_MULTIPROC_DIR environment variable
311
# before prometheus_client is imported.
312
# See https://prometheus.github.io/client_python/multiprocess/
313
- from prometheus_client import (CollectorRegistry, make_asgi_app,
313
+ from prometheus_client import (REGISTRY, CollectorRegistry, make_asgi_app,
314
multiprocess)
315
from prometheus_fastapi_instrumentator import Instrumentator
316
317
+ registry = REGISTRY
318
+
319
prometheus_multiproc_dir_path = os.getenv("PROMETHEUS_MULTIPROC_DIR", None)
320
if prometheus_multiproc_dir_path is not None:
321
logger.debug("vLLM to use %s as PROMETHEUS_MULTIPROC_DIR",
322
prometheus_multiproc_dir_path)
323
registry = CollectorRegistry()
324
multiprocess.MultiProcessCollector(registry)
323
- Instrumentator(
324
- excluded_handlers=[
325
- "/metrics",
326
- "/health",
327
- "/load",
328
- "/ping",
329
- "/version",
330
- "/server_info",
331
- ],
332
- registry=registry,
333
- ).add().instrument(app).expose(app)
334
-
335
- # Add prometheus asgi middleware to route /metrics requests
336
- metrics_route = Mount("/metrics", make_asgi_app(registry=registry))
337
- else:
338
- # Add prometheus asgi middleware to route /metrics requests
339
- metrics_route = Mount("/metrics", make_asgi_app())
325
+
326
+ Instrumentator(
327
+ excluded_handlers=[
328
+ "/metrics",
329
+ "/health",
330
+ "/load",
331
+ "/ping",
332
+ "/version",
333
+ "/server_info",
334
+ ],
335
+ registry=registry,
336
+ ).add().instrument(app).expose(app)
337
+
338
+ # Add prometheus asgi middleware to route /metrics requests
339
+ metrics_route = Mount("/metrics", make_asgi_app(registry=registry))
340
341
# Workaround for 307 Redirect for /metrics
342
metrics_route.path_regex = re.compile("^/metrics(?P<path>.*)$")