| 1 | ### Understand the alert |
| 2 | |
| 3 | The linux kernel contains queues where packets are stored after reception from a network interface controller before being processed by the next protocol stack. There is one netdev backlog queue per CPU core. netdev_max_backlog defines the maximum number of packets that can enter the queue. Queues fill up when an interface receives packets faster than kernel can process them. The default netdev_max_backlog value should be 1000. However this may not be enough in cases such as: |
| 4 | |
| 5 | - Multiple interfaces operating at 1Gbps, or even a single interface at 10Gbps. |
| 6 | |
| 7 | - Lower powered systems process very large amounts of network traffic. |
| 8 | |
| 9 | Netdata monitors the average number of dropped packets in the last minute due to exceeding the netdev backlog queue. |
| 10 | |
| 11 | ### Troubleshoot the alert |
| 12 | |
| 13 | - Increase the netdev_max_backlog value |
| 14 | |
| 15 | 1. Check your current value: |
| 16 | |
| 17 | ``` |
| 18 | root@netdata~ # sysctl net.core.netdev_max_backlog |
| 19 | net.core.netdev_max_backlog = 1000 |
| 20 | ``` |
| 21 | |
| 22 | 2. Try to increase it by a factor of 2. |
| 23 | |
| 24 | ``` |
| 25 | root@netdata~ # sysctl -w net.core.netdev_max_backlog=2000 |
| 26 | ``` |
| 27 | |
| 28 | 3. Verify the change and test with the same workload that triggered the alarm originally. |
| 29 | |
| 30 | ``` |
| 31 | root@netdata~ # sysctl net.core.netdev_max_backlog |
| 32 | net.core.netdev_max_backlog = 2000 |
| 33 | ``` |
| 34 | |
| 35 | 4. If this change works for your system, you could make it permanently. |
| 36 | |
| 37 | Bump this `net.core.netdev_max_backlog=2000` entry under `/etc/sysctl.conf`. |
| 38 | |
| 39 | 5. Reload the sysctl settings. |
| 40 | |
| 41 | ``` |
| 42 | root@netdata~ # sysctl -p |
| 43 | ``` |
| 44 |