| 1 | ### Understand the alert |
| 2 | |
| 3 | This alert indicates that your system is experiencing high IPv4 TCP socket utilization, specifically orphaned sockets. Orphaned connections are those not attached to any user file handle. When these connections exceed the limit, they are reset immediately. The warning state is triggered when the percentage of used orphan IPv4 TCP sockets exceeds 25%, and the critical state is triggered when the value exceeds 50%. |
| 4 | |
| 5 | ### Troubleshoot the alert |
| 6 | |
| 7 | - Check the current orphan socket usage |
| 8 | |
| 9 | To check the number of orphan sockets in your system, run the following command: |
| 10 | |
| 11 | ``` |
| 12 | cat /proc/sys/net/ipv4/tcp_max_orphans |
| 13 | ``` |
| 14 | |
| 15 | - Identify the processes causing high orphan socket usage |
| 16 | |
| 17 | To identify the processes causing high orphan socket usage, you can use the `ss` command: |
| 18 | |
| 19 | ``` |
| 20 | sudo ss -tan state time-wait state close-wait |
| 21 | ``` |
| 22 | |
| 23 | Look for connections with a large number of orphan sockets and investigate the related processes. |
| 24 | |
| 25 | - Increase the orphan socket limit |
| 26 | |
| 27 | If you need to increase the orphan socket limit to accommodate legitimate connections, you can update the value in the `/proc/sys/net/ipv4/tcp_max_orphans` file. Replace `{DESIRED_AMOUNT}` with the new limit: |
| 28 | |
| 29 | ``` |
| 30 | echo {DESIRED_AMOUNT} > /proc/sys/net/ipv4/tcp_max_orphans |
| 31 | ``` |
| 32 | |
| 33 | Consider the kernel's penalty factor for orphan sockets (usually 2x or 4x) when determining the appropriate limit. |
| 34 | |
| 35 | **Note**: Be cautious when making system changes and ensure you understand the implications of updating these settings. |
| 36 | |
| 37 | - Review and optimize application behavior |
| 38 | |
| 39 | Investigate the applications generating a high number of orphan sockets and consider optimizing their behavior. This may involve updating application settings or code to better manage network connections. |
| 40 | |
| 41 | - Monitor your system |
| 42 | |
| 43 | Keep an eye on your system's orphan socket usage, particularly during peak hours. Adjust the limit as needed to accommodate legitimate connections. |
| 44 | |
| 45 | ### Useful resources |
| 46 | |
| 47 | 1. [Network Sockets](https://en.wikipedia.org/wiki/Network_socket) |
| 48 | 2. [Linux-admins.com - Troubleshooting Out of Socket Memory](http://www.linux-admins.net/2013/01/troubleshooting-out-of-socket-memory.html) |