| 1 | ### Understand the alert |
| 2 | |
| 3 | This alert is triggered when the number of `sync()` system calls is greater than 6. The `sync()` system call writes any data buffered in memory out to disk, including modified superblocks, modified inodes, and delayed reads and writes. A higher number of `sync()` calls indicates that the system is often trying to flush buffered data to disk, which can cause performance issues. |
| 4 | |
| 5 | ### Troubleshoot the alert |
| 6 | |
| 7 | 1. Identify the process causing sync events |
| 8 | |
| 9 | Use `bpftrace` to identify which processes are causing the sync events. Make sure you have `bpftrace` installed on your system; if not, follow the instructions here: [Installing bpftrace](https://github.com/iovisor/bpftrace/blob/master/INSTALL.md) |
| 10 | |
| 11 | Run the `syncsnoop.bt` script from the `bpftrace` tools: |
| 12 | |
| 13 | ``` |
| 14 | sudo bpftrace /path/to/syncsnoop.bt |
| 15 | ``` |
| 16 | |
| 17 | This script will trace sync events and display the process ID (PID), process name, and the stack trace. |
| 18 | |
| 19 | 2. Analyze the output |
| 20 | |
| 21 | Focus on processes with a high number of sync events, and investigate whether you can optimize these processes or reduce their impact on the system. |
| 22 | |
| 23 | - Check if these processes are essential to system functionality. |
| 24 | - Look for potential bugs or misconfigurations that may trigger undue `sync()` calls. |
| 25 | - Consider modifying the process itself to reduce disk I/O or change how it handles write operations. |
| 26 | |
| 27 | 3. Monitor your system's I/O performance |
| 28 | |
| 29 | Keep an eye on overall I/O performance using tools like `iostat`, `iotop`, or `vmstat`. |
| 30 | |
| 31 | For example, you can use `iostat` to monitor disk I/O: |
| 32 | |
| 33 | ``` |
| 34 | iostat -xz 1 |
| 35 | ``` |
| 36 | |
| 37 | This command displays extended disk I/O statistics with a 1-second sampling interval. |
| 38 | |
| 39 | Check for high `await` values, which indicate the average time taken for I/O requests to be completed. Look for high `%util` values, representing the percentage of time the device was busy servicing requests. |
| 40 | |
| 41 | ### Useful resources |
| 42 | |
| 43 | 1. [sync man pages](https://man7.org/linux/man-pages/man2/sync.2.html) |
| 44 | 2. [bpftrace GitHub repository](https://github.com/iovisor/bpftrace) |
| 45 | 3. [syncsnoop example](https://github.com/iovisor/bpftrace/blob/master/tools/syncsnoop_example.txt) |
| 46 | 4. [iostat man pages](https://man7.org/linux/man-pages/man1/iostat.1.html) |