| 1 | ### Understand the alert |
| 2 | |
| 3 | This alert presents the average `I/O latency` over the last 10 seconds. `I/O latency` is the time that is required to complete a single I/O operation on a block device. |
| 4 | |
| 5 | This alert might indicate that your disk is under high load, or that the disk is slow. |
| 6 | |
| 7 | ### Troubleshoot the alert |
| 8 | |
| 9 | 1. Check per-process I/O usage: |
| 10 | |
| 11 | Use `iotop` to see the processes that are the main I/O consumers: |
| 12 | |
| 13 | ``` |
| 14 | sudo iotop |
| 15 | ``` |
| 16 | |
| 17 | If you don't have `iotop` installed, then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/) |
| 18 | |
| 19 | 2. Analyze the running processes: |
| 20 | |
| 21 | Investigate the top I/O consumers and determine if these processes are expected to consume that much I/O, or if there might be an issue with these processes. |
| 22 | |
| 23 | 3. Minimize the load by closing any unnecessary main consumer processes: |
| 24 | |
| 25 | If you find that any unnecessary or unexpected processes are heavily utilizing your disk, try stopping or closing those processes to reduce the load on the disk. Always double-check if the process you want to close is necessary. |
| 26 | |
| 27 | 4. Verify your disk health: |
| 28 | |
| 29 | Make sure your disk is not facing any hardware issues or failures. For this, you can use the `smartmontools` package, which contains the `smartctl` utility. If it's not installed, you can [install it](https://www.smartmontools.org/wiki/Download). |
| 30 | |
| 31 | To check the disk health, run: |
| 32 | |
| 33 | ``` |
| 34 | sudo smartctl -a /dev/sdX |
| 35 | ``` |
| 36 | |
| 37 | Replace `/dev/sdX` with the correct disk device identifier (for example, `/dev/sda`). |
| 38 | |
| 39 | 5. Consider upgrading your disk: |
| 40 | |
| 41 | If your disk consistently experiences high latency and you have already addressed any performance issues with the running processes, consider upgrading your disk to a faster drive (e.g., replace an HDD with an SSD). |
| 42 | |
| 43 | ### Useful resources |
| 44 | |
| 45 | 1. [iotop - Monitor Linux Disk I/O Activity](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/) |
| 46 | 2. [smartmontools - SMART monitoring tools](https://www.smartmontools.org/) |