| 1 | ### Understand the alert |
| 2 | |
| 3 | This alert, `load_cpu_number`, calculates the base trigger point for load average alarms, which helps identify when the system is overloaded. The alert checks the maximum number of CPUs in the system over the past 1 minute. If there is only one CPU, the trigger is set at 2. |
| 4 | |
| 5 | ### What does load average mean? |
| 6 | |
| 7 | The term `system load average` on a Linux machine measures the number of threads that are currently working and those waiting to work (CPU, disk, uninterruptible locks). In simpler terms, the load average measures the number of threads that aren't idle. |
| 8 | |
| 9 | ### What does overloaded mean? |
| 10 | |
| 11 | An overloaded system is when the demand on the system's resources (CPUs, disks, etc.) is higher than its capacity to handle tasks. This can lead to increased wait times, slower processing, and in worst cases, system crashes. |
| 12 | |
| 13 | ### Troubleshoot the alert |
| 14 | |
| 15 | 1. Determine the current load average on the system: |
| 16 | |
| 17 | Use the `uptime` command in the terminal to see the current load average: |
| 18 | ``` |
| 19 | uptime |
| 20 | ``` |
| 21 | |
| 22 | 2. Identify if the problem is CPU load or I/O load: |
| 23 | |
| 24 | Use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds) to get a report on system statistics: |
| 25 | |
| 26 | The `procs` column shows: |
| 27 | r: The number of runnable processes (running or waiting for run time). |
| 28 | b: The number of processes blocked waiting for I/O to complete. |
| 29 | |
| 30 | 3. Check per-process CPU/disk usage to find the top consumers: |
| 31 | |
| 32 | a. Use `top` to see the processes that are the main CPU consumers: |
| 33 | ``` |
| 34 | top -o +%CPU -i |
| 35 | ``` |
| 36 | |
| 37 | b. Use `iotop` to monitor Disk I/O usage (install it if not available): |
| 38 | ``` |
| 39 | sudo iotop |
| 40 | ``` |
| 41 | |
| 42 | 4. Minimize the load by closing any unnecessary main consumer processes. Double-check if the process you want to close is necessary. |
| 43 | |
| 44 | ### Useful resources |
| 45 | |
| 46 | 1. [Unix Load Average Part 1: How It Works](https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works) |
| 47 | 2. [Unix Load Average Part 2: Not Your Average Average](https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average) |
| 48 | 3. [Understanding Linux Process States](https://access.redhat.com/sites/default/files/attachments/processstates_20120831.pdf) |