@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 132 lines 7.37 KB
Rendered Raw
1 ---
2 name: kernel-microbenchmark
3 description: Build, debug, and interpret vLLM GPU kernel microbenchmarks for CUDA, ROCm/HIP, Triton, and CuteDSL, including CUPTI timing, correctness checks, generated-code inspection, multi-GPU measurements, and SOL sanity checks.
4 ---
5
6 # Kernel Microbenchmark
7
8 ## Workflow
9
10 1. Create an isolated repro or benchmark when the existing harness is noisy.
11 2. Check correctness before timing. Keep tolerances explicit.
12 3. Time only the operation under study. Exclude allocation, compilation, random
13 input generation, logging, and host-device transfers unless those are the
14 target.
15 4. Compare against a baseline and report enough metadata to reproduce the
16 result: GPU, dtype, shape, command, branch/commit, and relevant env vars.
17 5. Treat explanations as hypotheses until backed by an artifact: ablation,
18 generated PTX/SASS or AMD ISA, profiler output, or controlled benchmark.
19 6. If the result changes the conclusion, preserve the compact lesson in a note,
20 comment, benchmark table, or final summary. For experiments, a short
21 `Question / Change / Correctness / Result / Observation / Next` note is
22 usually enough.
23
24 ## Benchmark Defaults
25
26 - Use FlashInfer CUPTI timing by default, with CUDA graph and cold L2 cache:
27 `from flashinfer.testing import bench_gpu_time_with_cupti`.
28 - ROCm: CUPTI and FlashInfer do not exist. Use HIP graph replay over rotating
29 buffers whose total footprint exceeds the last-level cache (e.g. 256 MB on
30 MI355X), so every call reads cold from HBM, timed with HIP events — see
31 [benchmarks/graph_replay_benchmark.py](benchmarks/graph_replay_benchmark.py)
32 (platform-agnostic). Prefer this over `do_bench`, whose per-rep
33 launch/event overhead inflates small/decode-size kernels.
34 - For compute-heavy kernels, report TFLOPS with the FLOP formula in the
35 benchmark. For memory-heavy kernels, report estimated bytes moved and GB/s.
36 For mixed kernels, report the most honest metric available and call out the
37 caveats. TFLOPS and memory bandwidth should be computed from the theoretical
38 best for the operation, not from a particular kernel implementation. For
39 example, memory bandwidth should assume all data is read exactly once from
40 global memory.
41 - When comparing across shapes, prefer throughput metrics such as TFLOPS or
42 GB/s as the primary table columns; keep latency for absolute cost.
43 - If a result exceeds expected peak/SOL, first inspect units, FLOP/byte
44 formulas, skipped work, sparsity, caching, and whether the baseline is doing
45 the same operation.
46 - Force compilation/autotuning before measuring compiled kernels.
47 - Tune every arm of a comparison before trusting it. A variant's default or
48 inherited config can be far from its optimum, so conclusions drawn from
49 untuned variants — especially negative ones — are unreliable. A coarse
50 sweep of the cheap knobs at a couple of representative shapes is enough;
51 report the winning config with the numbers.
52 - Seed inputs when correctness comparisons matter.
53 - Keep metadata setup, plan construction, allocation, random input generation,
54 and logging outside the timed region unless that overhead is the experiment.
55
56 ## Sanity-Check Reference Numbers
57
58 Use these as rough reference points for large, well-shaped workloads, not as
59 gold standards, guaranteed peaks, or hard limits. Hardware SKU, clocks, shape,
60 precision conventions, and the FLOP/byte accounting can move the result. A
61 large gap is a prompt to investigate, not proof that a kernel is poor.
62
63 | Kernel regime | Hardware | Rough reference |
64 | --- | --- | ---: |
65 | Memory-bound, large batch | Blackwell | 6 TB/s |
66 | BF16 GEMM | Blackwell | 2 PFLOP/s |
67 | BF16 attention | B200 | 1.6 PFLOP/s |
68 | FP8 GEMM | Blackwell | 4 PFLOP/s |
69 | FP8 attention | B300 | 2.8 PFLOP/s |
70
71 The BF16 attention reference is approximately the 1613 TFLOP/s result reported
72 by the FlashAttention-4 paper. Compare kernels only with matching workload and
73 throughput conventions.
74
75 ## Multi-GPU Benchmarks
76
77 - State whether the run is local or multi-node and report the GPU topology,
78 world size, GPUs per node, collective backend, and relevant library versions.
79 - Compare like-for-like TP configurations. Report both per-rank and global
80 dimensions, and do not compare results from different TP sizes without an
81 explicit normalization or scaling question.
82 - Define the timed operation boundary before benchmarking. If the production
83 wrapper performs input staging, flag resets, generation barriers, padding,
84 or output copies, keep them in the timed region for an end-to-end comparison.
85 Use a separate, clearly labeled ablation for kernel-only timing.
86 - Check distributed correctness before timing. Seed each rank deliberately,
87 form the reference with the same collective semantics, and synchronize before
88 reading or comparing outputs.
89 - Use a device-side barrier immediately before each measured replay. Keep this
90 common synchronization outside the timed interval, but keep barriers required
91 by the candidate implementation inside it.
92 - With CUDA graphs, warm up before capture, coordinate capture across ranks,
93 rotate pointer-distinct graphs when cache reuse matters, and ensure every
94 collective is issued in the same order on every rank.
95 - Measure every rank and reduce each sample with `MAX`; report the median of
96 those per-sample maxima. A rank-local event time is not a distributed latency.
97 - Reset reusable symmetric-memory flags before each invocation and establish a
98 device-side generation barrier before peers may signal them. Otherwise a fast
99 rank can signal before a slow rank resets its flags, causing a lost arrival
100 and intermittent deadlock.
101 - Preserve symmetric-memory handles, CUDA graphs, streams, and graph outputs for
102 the full measurement lifetime. Allocate and rendezvous symmetric buffers in
103 identical order and with identical shapes on all ranks.
104 - Treat hangs and isolated millisecond outliers as synchronization bugs or rank
105 skew until disproven. Add stage markers, bounded synchronization checks, and
106 per-rank diagnostics before blaming compilation or kernel performance.
107 - For multi-node runs, record the scheduler allocation and verify the fabric
108 supports the required multicast or NVLink-domain assumptions. Do not describe
109 a two-node TP run as equivalent to a local NVLink-domain run without checking.
110 - Stabilize GPU clocks or run enough untimed work to reach a steady state.
111 Alternate candidate order so clock, thermal, and rank-skew effects are shared.
112 - ROCm: capturing ProcessGroupNCCL collectives in HIP graphs trips the NCCL
113 watchdog (HIP forbids querying an event recorded in a capturing stream;
114 the watchdog aborts the process). `TORCH_NCCL_BLOCKING_WAIT=1` before
115 `import torch` disables the watchdog — benchmark-only knob. Ask for backend
116 `"nccl"`; torch routes it to RCCL automatically (there is no `"rccl"`
117 backend string).
118
119 ## Included Examples
120
121 - Use [benchmarks/cupti_microbenchmark.py](benchmarks/cupti_microbenchmark.py)
122 as a minimal single-GPU FlashInfer CUPTI timing pattern.
123 - Use
124 [benchmarks/graph_replay_benchmark.py](benchmarks/graph_replay_benchmark.py)
125 as a graph-replay timing pattern over rotating buffers sized past the
126 last-level cache — the ROCm/HIP default, also usable on CUDA.
127 - Use
128 [benchmarks/multi_gpu_gemm_rs.py](benchmarks/multi_gpu_gemm_rs.py) as a
129 minimal distributed CUDA-graph timing pattern.
130
131 Adapt the operation, cases, work formula, and correctness tolerances rather
132 than copying either example unchanged.