@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 174 lines 6.28 KB
Rendered Raw
1 # gputrc2graph.py
2
3 This script processes NVIDIA Nsight Systems (`nsys`) GPU trace files
4 (`.nsys-rep`) with -t cuda tracing enabled, and generates kernel-level
5 summaries and visualizations of GPU and non-GPU time. It is useful for
6 profiling and analyzing nsys profile output.
7
8 ## Usage
9
10 ### Command-line Arguments
11
12 - `--in_file`
13 **(required)**
14 List of input files and their metadata. Each entry should be in the format:
15 `<nsys-rep>,<engine>,<model>,<elapsed_nonprofiled_sec>`
16 - `nsys-rep`: Path to the `.nsys-rep` file.
17 - `engine`: Engine name (e.g., `vllm`).
18 - `model`: Model name (e.g., `llama`, `gpt-oss`, `ds`).
19 - `elapsed_nonprofiled_sec`: Wall-clock runtime (in seconds) without
20 profiling. Specify `0` to use the elapsed time from the nsys-rep file
21 (this may inflate non-GPU time if actual runtime without profiling is
22 less). Multiple entries can be provided, separated by spaces.
23
24 - `--out_dir`
25 Output directory for the generated CSV and HTML files.
26 If not specified, results are saved in the current directory.
27
28 - `--title`
29 Title for the HTML chart/visualization.
30
31 - `--nsys_cmd`
32 Path to the `nsys` command.
33 Default: `nsys` (assumes it is in your PATH).
34 Use this if `nsys` is not in your system PATH.
35
36 ## Notes
37
38 - Make sure you have pandas installed.
39 - Make sure [nsys](https://developer.nvidia.com/nsight-systems/get-started) is installed, and specify the path to the `nsys` command with `--nsys_cmd` if it is not in your PATH.
40 - For more details on available engines and models, see the help string in
41 the script or run:
42
43 ```bash
44 python3 gputrc2graph.py --help
45 ```
46
47 ## Example 1: analyze a single profile
48
49 To analyze the GPU cycles for say, gpt-oss model with vLLM engine:
50
51 1. Run the following command to collect nsys profile, for vllm serve config.
52
53 ```bash
54 nsys profile -t cuda -o run1 -f true --trace-fork-before-exec=true \
55 --cuda-graph-trace=node --delay <DELAY> --duration <DURATION> \
56 vllm serve openai/gpt-oss-120b ...
57 ```
58
59 where:
60
61 - DELAY: how many seconds to delay nsys from collecting profiles, needed so
62 that profiles aren't captured till vllm server has come up and load
63 generation starts.
64 - DURATION: how many seconds for nsys profile to run before generating the
65 profile. This should be > the duration of the run.
66
67 2. Run again, this time without collecting the profile, and get the total run
68 time in seconds. This value will be used by the script to calculate the
69 CPU(non-GPU) seconds for the analysis.
70
71 3. Say the run elapsed time is 306 seconds, from step #2. Run script to
72 analyze:
73
74 ```bash
75 python3 gputrc2graph.py \
76 --in_file run1.nsys-rep,vllm,gpt-oss,306 \
77 --title "vLLM-gpt-oss profile"
78 ```
79
80 The command will produce 2 files for analysis:
81
82 - result.html: this categorizes kernel names into different categories in a
83 stacked bar chart.
84 - result.csv: shows how the kernel names are mapped to the different
85 categories.
86
87 ### HTML visualization with result.html
88
89 The html file shows the number of elapsed seconds due to different GPU
90 Substages or categories, which consist of moe_gemm (Mixture of Experts GEMM)
91 kernels the biggest category, at 148 seconds, followed by "attn" or attention
92 kernels. This lets the user prioritize the kernels to focus on for performance
93 optimizations.
94
95 ![Example GPU Trace Visualization](images/html.png)
96
97 There's also an appended data table underneath the bar chart for copying out to other post-processing tools.
98
99 ![Example GPU Trace Table](images/html_tbl.png)
100
101 ### Kernel to category mapping with result.csv
102
103 Suppose the user would like to focus on improving triton kernels. It's not the
104 biggest consumer of cycles at 9.74 sec but perhaps it hasn't been optimized.
105 The next step is to use the result.csv to dive into what the kernels are which
106 compose the triton kernel GPU cycles. The following image shows that
107 triton_poi_fused__to_copy_add_addmm_cat_.. kernel to be the biggest
108 contributor to GPU cycles.
109
110 ![Example GPU Trace csv](images/csv1.png)
111
112 ## Example 2: analyze multiple profiles
113
114 Suppose the user has multiple nsys trace files, captured for different models,
115 say llama and gpt-oss in this case, and wish to compare their GPU/non-GPU
116 time, something like the following command can be used.
117
118 ```bash
119 python3 gputrc2graph.py \
120 --in_file run1.nsys-rep,vllm,llama,100 run2.nsys-rep,vllm,gpt-oss,102 \
121 --out_dir results \
122 --title "Comparison of vLLM Models"
123 ```
124
125 The analysis process is similar to example 1 but now there will be multiple
126 stack bar charts that can be compared. The categories for the different
127 kernels will remain the same, so that it's easy to compare the GPU cycles for
128 the same categories.
129
130 Once a category is shown to have more cycles for one configuration than
131 another, the next step would be to use the csv file to see what kernels are
132 mapped into that category, and which kernels are taking the largest amount of
133 time which would cause a difference for the overall category.
134
135 ## Example 3: add new classification for a new model
136
137 To create a new engine DEF with model ABC, just add another json file in the same directory as
138 gputrc2graph.py with the same format as the other json files. The script will automatically pick up all the json files in the same directory as engine/model specifications.
139
140 Then, for this new model, suppose there are 4 kernels to be classified into "gemm" and "attn", where the gemm kernels
141 have names with "*H*" or "*I*" in them, and attn kernels have names with "*J*"
142 or "*K*" in them, just add another .json file in the same directory as
143 gputrc2graph.py with the same format as the other json files, like the following:
144
145 ```json
146 {
147 "DEF": {
148 "ABC": {
149 "H|I": "gemm",
150 "J|K": "attn",
151 "CUDA mem": "non-gpu-H_D_memops",
152 ".*": "misc"
153 }
154 }
155 }
156 ```
157
158 Each entry in the dictionary consists of:
159
160 - key: a regex used to classify the kernels
161 - value: the category to classify the kernels into.
162
163 The last 2 entries are common for all engine/models, consisting of CUDA memory
164 operations and a 'misc' for anything that's leftover and can't be classified.
165
166 When invoking gputrc2graph.py, specify a trace file with this new model/engine
167 like the following:
168
169 ```bash
170 --infile new.nsys-rep,DEF,ABC,<runtime>
171 ```
172
173 If the engine_DEF.json file already exists, just add the model as a new node in
174 the existing engine file, after the other models.