| 1 | ### Understand the alert |
| 2 | |
| 3 | This alert checks the health status of the Consul cluster regarding its autopilot functionality. If you receive this alert, it means that the Consul datacenter is experiencing issues, and its health status has been reported as `unhealthy` by the Consul server. |
| 4 | |
| 5 | ### What is Consul autopilot? |
| 6 | |
| 7 | Consul's autopilot feature provides automatic management and stabilization features for Consul server clusters, ensuring that the clusters remain in a healthy state. These features include server health monitoring, automatic dead server reaping, and stable server introduction. |
| 8 | |
| 9 | ### What does unhealthy mean? |
| 10 | |
| 11 | An unhealthy Consul cluster could experience issues regarding its operations, services, leader elections, and cluster consistency. In this alert scenario, the cluster health functionality is not working correctly, and it could lead to stability and performance problems. |
| 12 | |
| 13 | ### Troubleshoot the alert |
| 14 | |
| 15 | Here are some steps to troubleshoot the consul_autopilot_health_status alert: |
| 16 | |
| 17 | 1. Check the logs of the Consul server to identify any error messages or warning signs. The logs will often provide insights into the underlying problems. |
| 18 | |
| 19 | ``` |
| 20 | journalctl -u consul |
| 21 | ``` |
| 22 | |
| 23 | 2. Inspect the Consul health status using the Consul CLI or API: |
| 24 | |
| 25 | ``` |
| 26 | consul operator autopilot get-config |
| 27 | ``` |
| 28 | |
| 29 | Using the Consul HTTP API: |
| 30 | ``` |
| 31 | curl http://<consul_server>:8500/v1/operator/autopilot/health |
| 32 | ``` |
| 33 | |
| 34 | 3. Verify the configuration of Consul servers, check the `retry_join` and addresses of the Consul servers in the configuration file: |
| 35 | |
| 36 | ``` |
| 37 | cat /etc/consul.d/consul.hcl | grep retry_join |
| 38 | ``` |
| 39 | |
| 40 | 4. Ensure that there is a sufficient number of Consul servers and that they are healthy. The `consul members` command will show the status of cluster members: |
| 41 | |
| 42 | ``` |
| 43 | consul members |
| 44 | ``` |
| 45 | |
| 46 | 5. Check the network connectivity between Consul servers by running network diagnostics like ping and traceroute. |
| 47 | |
| 48 | 6. Review Consul documentation to gain a deeper understanding of the autopilot health issues and potential configuration problems. |
| 49 | |
| 50 | |
| 51 | ### Useful resources |
| 52 | |
| 53 | - [Consul CLI reference](https://www.consul.io/docs/commands) |