| 1 | ### Understand the alert |
| 2 | |
| 3 | This alarm calculates the system `load average` (CPU and I/O demand) over the period of five minutes. If you receive this alarm, it means that your system is "overloaded." |
| 4 | |
| 5 | The alert gets raised into warning if the metric is 4 times the expected value and cleared if the value is 3.5 times the expected value. |
| 6 | |
| 7 | For further information on how our alerts are calculated, please have a look at our [Documentation](/src/health/REFERENCE.md#expressions). |
| 8 | |
| 9 | |
| 10 | ### What does "load average" mean? |
| 11 | |
| 12 | The term `system load average` on a Linux machine, measures the **number of threads that are currently working and those waiting to work** (CPU, disk, uninterruptible locks). So simply stated: **System load average measures the number of threads that aren't idle.** |
| 13 | |
| 14 | ### What does "overloaded" mean? |
| 15 | |
| 16 | Let's look at a single core CPU system and think of its core count as car lanes on a bridge. A car represents a process in this example: |
| 17 | |
| 18 | - On a 0.5 load average, the traffic on the bridge is fine, it is at 50% of its capacity. |
| 19 | - If the load average is at 1, then the bridge is full, and it is utilized 100%. |
| 20 | - If the load average gets to 2 (remember we are on a single core machine), it means that there is one car lane that is passing the bridge. However, there is **another** full car lane that waits to pass the bridge. |
| 21 | |
| 22 | So this is how you can imagine CPU load, but keep in mind that `load average` counts also I/O demand, so there is an analogous example there. |
| 23 | |
| 24 | ### Useful resources |
| 25 | |
| 26 | 1. [UNIX Load Average Part 1: How It Works](https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works) |
| 27 | 2. [UNIX Load Average Part 2: Not Your Average Average](https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average) |
| 28 | 3. [Understanding Linux CPU Load](https://scoutapm.com/blog/understanding-load-averages) |
| 29 | 4. [Linux Load Averages: Solving the Mystery](https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html) |
| 30 | 5. [Understanding Linux Process States](https://access.redhat.com/sites/default/files/attachments/processstates_20120831.pdf) |
| 31 | |
| 32 | |
| 33 | ### Troubleshoot the alert |
| 34 | |
| 35 | - Determine if the problem is CPU or I/O bound |
| 36 | |
| 37 | First you need to check if you are running on a CPU load or an I/O load problem. |
| 38 | |
| 39 | 1. To get a report about your system statistics, use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds): |
| 40 | |
| 41 | The `procs` column, shows: |
| 42 | r: The number of runnable processes (running or waiting for run time). |
| 43 | b: The number of processes blocked waiting for I/O to complete. |
| 44 | |
| 45 | 2. List your currently running processes using the `ps` command: |
| 46 | |
| 47 | The `grep` command will fetch the processes that their state code starts either with R (running or runnable (on run queue)) or D(uninterruptible sleep (usually IO)). |
| 48 | |
| 49 | 3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary. |
| 50 | |
| 51 | - Check per-process CPU/disk usage to find the top consumers |
| 52 | |
| 53 | 1. To see the processes that are the main CPU consumers, use the task manager program `top` like this: |
| 54 | |
| 55 | ``` |
| 56 | top -o +%CPU -i |
| 57 | ``` |
| 58 | |
| 59 | 2. Use `iotop`: |
| 60 | `iotop` is a useful tool, similar to `top`, used to monitor Disk I/O usage, if you don't have it, then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/) |
| 61 | ``` |
| 62 | sudo iotop |
| 63 | ``` |
| 64 | |
| 65 | 3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary. |
| 66 |