A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 08:15 UTC
Doc: Improve benchmark documentation (#9927)
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
Rafael Vasquez committed
Nov 7, 2024 at 02:50 UTC
d7263a1bb837648bec67d99ed35db56c58832d3f
4 files changed
+38
-27
docs/source/dev/profiling/profiling_index.rst
+3
-2
@@ -1,5 +1,6 @@
1
-Profiling vLLM
2
-=================================
1
+==============
2
+Profiling vLLM
3
+==============
4
5
We support tracing vLLM workers using the ``torch.profiler`` module. You can enable tracing by setting the ``VLLM_TORCH_PROFILER_DIR`` environment variable to the directory where you want to save the traces: ``VLLM_TORCH_PROFILER_DIR=/mnt/traces/``
6
docs/source/index.rst
+2
-2
@@ -126,9 +126,9 @@ Documentation
126
127
.. toctree::
128
:maxdepth: 1
129
- :caption: Performance benchmarks
129
+ :caption: Performance
130
131
- performance_benchmark/benchmarks
131
+ performance/benchmarks
132
133
.. toctree::
134
:maxdepth: 2
docs/source/performance/benchmarks.rst
new
+33
@@ -0,0 +1,33 @@
1
+.. _benchmarks:
2
+
3
+================
4
+Benchmark Suites
5
+================
6
+
7
+vLLM contains two sets of benchmarks:
8
+
9
++ :ref:`Performance benchmarks <performance_benchmarks>`
10
++ :ref:`Nightly benchmarks <nightly_benchmarks>`
11
+
12
+
13
+.. _performance_benchmarks:
14
+
15
+Performance Benchmarks
16
+----------------------
17
+
18
+The performance benchmarks are used for development to confirm whether new changes improve performance under various workloads. They are triggered on every commit with both the ``perf-benchmarks`` and ``ready`` labels, and when a PR is merged into vLLM.
19
+
20
+The latest performance results are hosted on the public `vLLM Performance Dashboard <https://perf.vllm.ai>`_.
21
+
22
+More information on the performance benchmarks and their parameters can be found `here <https://github.com/vllm-project/vllm/blob/main/.buildkite/nightly-benchmarks/performance-benchmarks-descriptions.md>`__.
23
+
24
+.. _nightly_benchmarks:
25
+
26
+Nightly Benchmarks
27
+------------------
28
+
29
+These compare vLLM's performance against alternatives (``tgi``, ``trt-llm``, and ``lmdeploy``) when there are major updates of vLLM (e.g., bumping up to a new version). They are primarily intended for consumers to evaluate when to choose vLLM over other options and are triggered on every commit with both the ``perf-benchmarks`` and ``nightly-benchmarks`` labels.
30
+
31
+The latest nightly benchmark results are shared in major release blog posts such as `vLLM v0.6.0 <https://blog.vllm.ai/2024/09/05/perf-update.html>`_.
32
+
33
+More information on the nightly benchmarks and their parameters can be found `here <https://github.com/vllm-project/vllm/blob/main/.buildkite/nightly-benchmarks/nightly-descriptions.md>`__.
\ No newline at end of file
docs/source/performance_benchmark/benchmarks.rst
deleted
-23
@@ -1,23 +0,0 @@
1
-.. _benchmarks:
2
-
3
-Benchmark suites of vLLM
4
-========================
5
-
6
-
7
-
8
-vLLM contains two sets of benchmarks:
9
-
10
-+ **Performance benchmarks**: benchmark vLLM's performance under various workloads at a high frequency (when a pull request (PR for short) of vLLM is being merged). See `vLLM performance dashboard <https://perf.vllm.ai>`_ for the latest performance results.
11
-
12
-+ **Nightly benchmarks**: compare vLLM's performance against alternatives (tgi, trt-llm, and lmdeploy) when there are major updates of vLLM (e.g., bumping up to a new version). The latest results are available in the `vLLM GitHub README <https://github.com/vllm-project/vllm/blob/main/README.md>`_.
13
-
14
-
15
-Trigger a benchmark
16
--------------------
17
-
18
-The performance benchmarks and nightly benchmarks can be triggered by submitting a PR to vLLM, and label the PR with `perf-benchmarks` and `nightly-benchmarks`.
19
-
20
-
21
-.. note::
22
-
23
- Please refer to `vLLM performance benchmark descriptions <https://github.com/vllm-project/vllm/blob/main/.buildkite/nightly-benchmarks/performance-benchmarks-descriptions.md>`_ and `vLLM nightly benchmark descriptions <https://github.com/vllm-project/vllm/blob/main/.buildkite/nightly-benchmarks/nightly-descriptions.md>`_ for detailed descriptions on benchmark environment, workload and metrics.