@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / d7263a1bb8
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 08:15 UTC

Doc: Improve benchmark documentation (#9927)

Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>

Rafael Vasquez committed Nov 7, 2024 at 02:50 UTC d7263a1bb837648bec67d99ed35db56c58832d3f
4 files changed +38 -27
docs/source/dev/profiling/profiling_index.rst
+3 -2
@@ -1,5 +1,6 @@
1 -Profiling vLLM
2 -=================================
1 +==============
2 +Profiling vLLM
3 +==============
4
5 We support tracing vLLM workers using the ``torch.profiler`` module. You can enable tracing by setting the ``VLLM_TORCH_PROFILER_DIR`` environment variable to the directory where you want to save the traces: ``VLLM_TORCH_PROFILER_DIR=/mnt/traces/``
6
docs/source/index.rst
+2 -2
@@ -126,9 +126,9 @@ Documentation
126
127 .. toctree::
128 :maxdepth: 1
129 - :caption: Performance benchmarks
129 + :caption: Performance
130
131 - performance_benchmark/benchmarks
131 + performance/benchmarks
132
133 .. toctree::
134 :maxdepth: 2
docs/source/performance/benchmarks.rst new
+33
@@ -0,0 +1,33 @@
1 +.. _benchmarks:
2 +
3 +================
4 +Benchmark Suites
5 +================
6 +
7 +vLLM contains two sets of benchmarks:
8 +
9 ++ :ref:`Performance benchmarks <performance_benchmarks>`
10 ++ :ref:`Nightly benchmarks <nightly_benchmarks>`
11 +
12 +
13 +.. _performance_benchmarks:
14 +
15 +Performance Benchmarks
16 +----------------------
17 +
18 +The performance benchmarks are used for development to confirm whether new changes improve performance under various workloads. They are triggered on every commit with both the ``perf-benchmarks`` and ``ready`` labels, and when a PR is merged into vLLM.
19 +
20 +The latest performance results are hosted on the public `vLLM Performance Dashboard <https://perf.vllm.ai>`_.
21 +
22 +More information on the performance benchmarks and their parameters can be found `here <https://github.com/vllm-project/vllm/blob/main/.buildkite/nightly-benchmarks/performance-benchmarks-descriptions.md>`__.
23 +
24 +.. _nightly_benchmarks:
25 +
26 +Nightly Benchmarks
27 +------------------
28 +
29 +These compare vLLM's performance against alternatives (``tgi``, ``trt-llm``, and ``lmdeploy``) when there are major updates of vLLM (e.g., bumping up to a new version). They are primarily intended for consumers to evaluate when to choose vLLM over other options and are triggered on every commit with both the ``perf-benchmarks`` and ``nightly-benchmarks`` labels.
30 +
31 +The latest nightly benchmark results are shared in major release blog posts such as `vLLM v0.6.0 <https://blog.vllm.ai/2024/09/05/perf-update.html>`_.
32 +
33 +More information on the nightly benchmarks and their parameters can be found `here <https://github.com/vllm-project/vllm/blob/main/.buildkite/nightly-benchmarks/nightly-descriptions.md>`__.
\ No newline at end of file
docs/source/performance_benchmark/benchmarks.rst deleted
-23
@@ -1,23 +0,0 @@
1 -.. _benchmarks:
2 -
3 -Benchmark suites of vLLM
4 -========================
5 -
6 -
7 -
8 -vLLM contains two sets of benchmarks:
9 -
10 -+ **Performance benchmarks**: benchmark vLLM's performance under various workloads at a high frequency (when a pull request (PR for short) of vLLM is being merged). See `vLLM performance dashboard <https://perf.vllm.ai>`_ for the latest performance results.
11 -
12 -+ **Nightly benchmarks**: compare vLLM's performance against alternatives (tgi, trt-llm, and lmdeploy) when there are major updates of vLLM (e.g., bumping up to a new version). The latest results are available in the `vLLM GitHub README <https://github.com/vllm-project/vllm/blob/main/README.md>`_.
13 -
14 -
15 -Trigger a benchmark
16 --------------------
17 -
18 -The performance benchmarks and nightly benchmarks can be triggered by submitting a PR to vLLM, and label the PR with `perf-benchmarks` and `nightly-benchmarks`.
19 -
20 -
21 -.. note::
22 -
23 - Please refer to `vLLM performance benchmark descriptions <https://github.com/vllm-project/vllm/blob/main/.buildkite/nightly-benchmarks/performance-benchmarks-descriptions.md>`_ and `vLLM nightly benchmark descriptions <https://github.com/vllm-project/vllm/blob/main/.buildkite/nightly-benchmarks/nightly-descriptions.md>`_ for detailed descriptions on benchmark environment, workload and metrics.