| 1 | # Benchmarks |
| 2 | |
| 3 | This directory used to contain vLLM's benchmark scripts and utilities for performance testing and evaluation. |
| 4 | |
| 5 | ## Contents |
| 6 | |
| 7 | - **Serving benchmarks**: Scripts for testing online inference performance (latency, throughput) |
| 8 | - **Throughput benchmarks**: Scripts for testing offline batch inference performance |
| 9 | - **Specialized benchmarks**: Tools for testing specific features like structured output, prefix caching, long document QA, request prioritization, and multi-modal inference |
| 10 | - **Dataset utilities**: Framework for loading and sampling from various benchmark datasets (ShareGPT, HuggingFace datasets, synthetic data, etc.) |
| 11 | |
| 12 | ## Usage |
| 13 | |
| 14 | For detailed usage instructions, examples, and dataset information, see the [Benchmark CLI documentation](https://docs.vllm.ai/en/latest/benchmarking/cli/#benchmark-cli). |
| 15 | |
| 16 | For full CLI reference see: |
| 17 | |
| 18 | - <https://docs.vllm.ai/en/latest/cli/bench/latency.html> |
| 19 | - <https://docs.vllm.ai/en/latest/cli/bench/serve.html> |
| 20 | - <https://docs.vllm.ai/en/latest/cli/bench/throughput.html> |