| 1 | ### Understand the alert |
| 2 | |
| 3 | This alarm calculates the system `load average` (`CPU` and `I/O` demand) over the period of one minute. If you receive this alarm, it means that your system is `overloaded`. |
| 4 | |
| 5 | ### What does "load average" mean? |
| 6 | |
| 7 | The term `system load average` on a Linux machine, measures the **number of threads that are currently working and those waiting to work** (CPU, disk, uninterruptible locks). So simply stated: **System load average measures the number of threads that aren't idle.** |
| 8 | |
| 9 | ### What does "overloaded" mean? |
| 10 | |
| 11 | Let's look at a single core CPU system and think of its core count as car lanes on a bridge. A car represents a process in this example: |
| 12 | |
| 13 | - On a 0.5 load average, the traffic on the bridge is fine, it is at 50% of its capacity. |
| 14 | - If the load average is at 1, then the bridge is full, and it is utilized 100%. |
| 15 | - If the load average gets to 2 (remember we are on a single core machine), it means that there is one car lane that is passing the bridge. However, there is **another** full car lane that waits to pass the bridge. |
| 16 | |
| 17 | So this is how you can imagine CPU load, but keep in mind that `load average` counts also I/O demand, so there is an analogous example there. |
| 18 | |
| 19 | ### Troubleshoot the alert |
| 20 | |
| 21 | - Determine if the problem is CPU load or I/O load |
| 22 | |
| 23 | To get a report about your system statistics, use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds): |
| 24 | |
| 25 | The `procs` column, shows: |
| 26 | r: The number of runnable processes (running or waiting for run time). |
| 27 | b: The number of processes blocked waiting for I/O to complete. |
| 28 | |
| 29 | - Check per-process CPU/disk usage to find the top consumers |
| 30 | |
| 31 | 1. To see the processes that are the main CPU consumers, use the task manager program `top` like this: |
| 32 | |
| 33 | ``` |
| 34 | top -o +%CPU -i |
| 35 | ``` |
| 36 | |
| 37 | 2. Use `iotop`: |
| 38 | `iotop` is a useful tool, similar to `top`, used to monitor Disk I/O usage, if you don't have it, then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/) |
| 39 | ``` |
| 40 | sudo iotop |
| 41 | ``` |
| 42 | |
| 43 | 3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary. |
| 44 | |
| 45 | ### Useful resources |
| 46 | |
| 47 | 1. [UNIX Load Average Part 1: How It Works](https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works) |
| 48 | 2. [UNIX Load Average Part 2: Not Your Average Average](https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average) |
| 49 | 3. [Understanding Linux CPU Load](https://scoutapm.com/blog/understanding-load-averages) |
| 50 | 4. [Linux Load Averages: Solving the Mystery](https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html) |
| 51 | 5. [Understanding Linux Process States](https://access.redhat.com/sites/default/files/attachments/processstates_20120831.pdf) |