A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 12, 06:00 UTC
[Doc] fix heading levels (#29783)
Signed-off-by: KKKZOZ <kkkzoz@qq.com>
Liu Jinyi committed
Dec 1, 2025 at 22:49 UTC
f5516039c5cc7ddb27a62f7e276a388723cc8542
1 file changed
+2
-2
benchmarks/auto_tune/README.md
+2
-2
@@ -83,7 +83,7 @@ MIN_CACHE_HIT_PCT=0
83
MAX_LATENCY_ALLOWED_MS=100000000000 # A very large number
84
```
85
86
-#### 2. Maximize Throughput with a Latency Requirement
86
+### 2. Maximize Throughput with a Latency Requirement
87
88
- **Goal**: Find the best server parameters when P99 end-to-end latency must be below 500ms.
89
- **Configuration**:
@@ -96,7 +96,7 @@ MIN_CACHE_HIT_PCT=0
96
MAX_LATENCY_ALLOWED_MS=500
97
```
98
99
-#### 3. Maximize Throughput with Prefix Caching and Latency Requirements
99
+### 3. Maximize Throughput with Prefix Caching and Latency Requirements
100
101
- **Goal**: Find the best server parameters assuming a 60% prefix cache hit rate and a latency requirement of 500ms.
102
- **Configuration**: