| 1 | ### Understand the alert |
| 2 | |
| 3 | This alert triggers when there is a `leadership transition` in the `Consul` service mesh. If you receive this alert, it means that server `${label:node_name}` in datacenter `${label:datacenter}` has become the new leader. |
| 4 | |
| 5 | ### What does consul_raft_leadership_transitions mean? |
| 6 | |
| 7 | Consul is a service mesh solution that provides service discovery, configuration, and segmentation functionality. It uses the Raft consensus algorithm to maintain a consistent data state across the cluster. A leadership transition occurs when the current leader node loses its leadership status and a different node takes over. |
| 8 | |
| 9 | ### What causes leadership transitions? |
| 10 | |
| 11 | Leadership transitions in Consul can be caused by various reasons, such as: |
| 12 | |
| 13 | 1. Network communication issues between the nodes. |
| 14 | 2. High resource utilization on the leader node, causing it to miss heartbeat messages. |
| 15 | 3. Nodes crashing or being intentionally shut down. |
| 16 | 4. A forced leadership transition triggered by an operator. |
| 17 | |
| 18 | Frequent leadership transitions may lead to service disruptions, increased latency, and reduced availability. Therefore, it's essential to identify and resolve the root cause promptly. |
| 19 | |
| 20 | ### Troubleshoot the alert |
| 21 | |
| 22 | 1. Check the Consul logs for indications of network issues or node failures: |
| 23 | |
| 24 | ``` |
| 25 | journalctl -u consul.service |
| 26 | ``` |
| 27 | Alternatively, you can check the Consul log file, which is usually located at `/var/log/consul/consul.log`. |
| 28 | |
| 29 | 2. Inspect the health and status of the Consul cluster using the `consul members` command: |
| 30 | |
| 31 | ``` |
| 32 | consul members |
| 33 | ``` |
| 34 | This command lists all cluster members and their roles, including the new leader node. |
| 35 | |
| 36 | 3. Determine if there's high resource usage on the affected nodes by monitoring CPU, memory, and disk usage: |
| 37 | |
| 38 | ``` |
| 39 | top |
| 40 | ``` |
| 41 | |
| 42 | 4. Examine network connectivity between nodes using tools like `ping`, `traceroute`, or `mtr`. |
| 43 | |
| 44 | 5. If the transitions are forced by operators, review the changes made and their impact on the cluster. |
| 45 | |
| 46 | 6. Consider increasing the heartbeat timeout configuration to allow the leader more time to respond, especially if high resource usage is causing frequent leadership transitions. |
| 47 | |
| 48 | 7. Review Consul's documentation on [consensus and leadership](https://developer.hashicorp.com/consul/docs/architecture/consensus) and [operation and maintenance](https://developer.hashicorp.com/consul/docs/guides) to gain insights into best practices and ways to mitigate leadership transitions. |
| 49 | |
| 50 | ### Useful resources |
| 51 | |
| 52 | 1. [Consul: Service Mesh Overview](https://www.consul.io/docs/intro) |
| 53 | 2. [Consul: Understanding Consensus and Leadership](https://developer.hashicorp.com/consul/docs/architecture/consensus) |
| 54 | 3. [Consul: Installation, Configuration, and Maintenance](https://developer.hashicorp.com/consul/docs/guides) |