| 1 | ### Understand the alert |
| 2 | |
| 3 | This alert is raised when the number of unhandled messages in the last minute, monitored by the Netdata Agent, is too high. It indicates that many messages were not delivered due to connections with `clean_session=true` in a VerneMQ messaging system. |
| 4 | |
| 5 | ### What does clean_session=true mean? |
| 6 | |
| 7 | In MQTT, `clean_session=true` means that the client doesn't want to store any session state on the broker for the duration of its connection. When the session is terminated, all subscriptions and messages are deleted. The broker won't store any messages or send any missed messages once the client reconnects. |
| 8 | |
| 9 | ### What are VerneMQ unhandled messages? |
| 10 | |
| 11 | Unhandled messages are messages that cannot be delivered to subscribers due to connection issues, protocol limitations, or session configurations. These messages are often related to clients' settings for `clean_session=true`, which means they don't store any session state on the broker. |
| 12 | |
| 13 | ### Troubleshoot the alert |
| 14 | |
| 15 | - Identify clients causing unhandled messages |
| 16 | |
| 17 | One way to find the clients causing unhandled messages is by analyzing the VerneMQ log files. Look for warning or error messages related to undelivered messages or clean sessions. The log files are typically located in `/var/log/vernemq/`. |
| 18 | |
| 19 | - Check clients' clean_session settings |
| 20 | |
| 21 | Review your MQTT clients' configurations to verify if they have `clean_session=true`. Consider changing the setting to `clean_session=false` if you want the broker to store session state and send missed messages upon reconnection. |
| 22 | |
| 23 | - Monitor VerneMQ statistics |
| 24 | |
| 25 | Use the following command to see an overview of the VerneMQ statistics: |
| 26 | |
| 27 | ``` |
| 28 | vmq-admin metrics show |
| 29 | ``` |
| 30 | |
| 31 | Look for metrics related to dropped or unhandled messages, such as `gauge.queue_message_unhandled`. |
| 32 | |
| 33 | - Examine your system resources |
| 34 | |
| 35 | High unhandled message rates can also be a result of insufficient system resources. Check your system resources (CPU, memory, disk usage) and consider upgrading if necessary. |
| 36 | |
| 37 | ### Useful resources |
| 38 | |
| 39 | 1. [VerneMQ - An MQTT Broker](https://vernemq.com/) |
| 40 | 2. [VerneMQ Documentation: Monitoring & Metrics](https://docs.vernemq.com/monitoring/) |
| 41 | 3. [Understanding MQTT Clean Sessions, Queuing, Retained Messages and QoS](https://www.hivemq.com/blog/mqtt-essentials-part-7-persistent-session-queuing-messages/) |