@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 9afa014552
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 07:00 UTC

Add small example to metrics.rst (#10550)

Michael Goin committed Nov 21, 2024 at 18:43 UTC 9afa01455237892c878bb2810912c487d66149a9
1 file changed +26 -1
docs/source/serving/metrics.rst
+26 -1
@@ -2,9 +2,34 @@ Production Metrics
2 ==================
3
4 vLLM exposes a number of metrics that can be used to monitor the health of the
5 -system. These metrics are exposed via the `/metrics` endpoint on the vLLM
5 +system. These metrics are exposed via the ``/metrics`` endpoint on the vLLM
6 OpenAI compatible API server.
7
8 +You can start the server using Python, or using [Docker](deploying_with_docker.rst):
9 +
10 +.. code-block:: console
11 +
12 + $ vllm serve unsloth/Llama-3.2-1B-Instruct
13 +
14 +Then query the endpoint to get the latest metrics from the server:
15 +
16 +.. code-block:: console
17 +
18 + $ curl http://0.0.0.0:8000/metrics
19 +
20 + # HELP vllm:iteration_tokens_total Histogram of number of tokens per engine_step.
21 + # TYPE vllm:iteration_tokens_total histogram
22 + vllm:iteration_tokens_total_sum{model_name="unsloth/Llama-3.2-1B-Instruct"} 0.0
23 + vllm:iteration_tokens_total_bucket{le="1.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
24 + vllm:iteration_tokens_total_bucket{le="8.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
25 + vllm:iteration_tokens_total_bucket{le="16.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
26 + vllm:iteration_tokens_total_bucket{le="32.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
27 + vllm:iteration_tokens_total_bucket{le="64.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
28 + vllm:iteration_tokens_total_bucket{le="128.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
29 + vllm:iteration_tokens_total_bucket{le="256.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
30 + vllm:iteration_tokens_total_bucket{le="512.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
31 + ...
32 +
33 The following metrics are exposed:
34
35 .. literalinclude:: ../../../vllm/engine/metrics.py