| 1 | ### Understand the alert |
| 2 | |
| 3 | This alert indicates that VerneMQ, an MQTT broker, is experiencing issues with inter-node message delivery within a clustered environment. The Netdata Agent calculates the amount of traffic dropped during communication with cluster nodes in the last minute. If you receive this alert, it means that the outgoing cluster buffer is full and some messages cannot be delivered. |
| 4 | |
| 5 | ### What does dropped messages mean? |
| 6 | |
| 7 | Dropped messages occur when the outgoing cluster buffer becomes full, and VerneMQ cannot deliver messages between its nodes. This can happen due to a remote node being down or unreachable, causing the buffer to fill up and preventing efficient message delivery. |
| 8 | |
| 9 | ### Troubleshoot the alert |
| 10 | |
| 11 | 1. Check the connectivity and status of cluster nodes |
| 12 | |
| 13 | Verify that all cluster nodes are up, running and reachable. Use `vmq-admin cluster show` to get an overview of the cluster nodes and their connectivity status. |
| 14 | |
| 15 | ``` |
| 16 | vmq-admin cluster show |
| 17 | ``` |
| 18 | |
| 19 | 2. Investigate logs for any errors or warnings |
| 20 | |
| 21 | Inspect the logs of the VerneMQ node(s) for any errors or warning messages. This can provide insight into any potential problems related to the cluster or network. |
| 22 | |
| 23 | ``` |
| 24 | sudo journalctl -u vernemq |
| 25 | ``` |
| 26 | |
| 27 | 3. Increase the buffer size |
| 28 | |
| 29 | If the issue persists, consider increasing the buffer size. Adjust the `outgoing_clustering_buffer_size` value in the `vernemq.conf` file. |
| 30 | |
| 31 | ``` |
| 32 | outgoing_clustering_buffer_size = <new_buffer_size> |
| 33 | ``` |
| 34 | |
| 35 | Replace `<new_buffer_size>` with a larger value, for example, doubling the current buffer size. After updating the configuration, restart the VerneMQ service to apply the changes. |
| 36 | |
| 37 | ``` |
| 38 | sudo systemctl restart vernemq |
| 39 | ``` |
| 40 | |
| 41 | 4. Monitor the dropped messages |
| 42 | |
| 43 | Continue to monitor the dropped messages using Netdata, and check if the issue is resolved after increasing the buffer size. |
| 44 | |
| 45 | ### Useful resources |
| 46 | |
| 47 | 1. [VerneMQ Documentation - Clustering](https://vernemq.com/docs/clustering/) |
| 48 | 2. [VerneMQ Logging and Monitoring](https://docs.vernemq.com/monitoring-vernemq/logging) |
| 49 | 3. [Managing VerneMQ Configuration](https://docs.vernemq.com/configuration/) |