| 1 | This directory contains tuned configurations for different settings of the fused_moe kernel. |
| 2 | For different settings of |
| 3 | - E (number of experts) |
| 4 | - N (intermediate size) |
| 5 | - device_name (torch.cuda.get_device_name()) |
| 6 | the JSON file contains a mapping from M (batch size) to the chosen configuration. |
| 7 | |
| 8 | The example configurations provided are for the Mixtral model for TP2 on H100 |
| 9 | and TP4 on A100. Mixtral has intermediate size N = 14336, i.e. for TP2 we have |
| 10 | N = 7168 and for TP4 we have N = 3584. |
| 11 | |
| 12 | See `benchmark/kernels/benchmark_moe.py` on how to generate these config files. |