| 1 | ### Understand the alert |
| 2 | |
| 3 | This alert is related to Kubernetes Kubelet token requests. It monitors the number of failed `Token()` requests to an alternate token source. If you receive this alert, it means that your system is experiencing an increased rate of token request failures. |
| 4 | |
| 5 | ### What does a token request in Kubernetes mean? |
| 6 | |
| 7 | In Kubernetes, tokens are used for authentication purposes when making requests to the API server. The Kubelet uses tokens to authenticate itself when it needs to access cluster information or manage resources on the API server. |
| 8 | |
| 9 | ### Troubleshoot the alert |
| 10 | |
| 11 | - Investigate the reason behind the failed token requests |
| 12 | |
| 13 | 1. Check the Kubelet logs for any error messages or warnings related to the token requests. You can use the following command to view the logs: |
| 14 | |
| 15 | ``` |
| 16 | journalctl -u kubelet |
| 17 | ``` |
| 18 | |
| 19 | Look for any entries related to `Token()` request failures or authentication issues. |
| 20 | |
| 21 | 2. Verify the alternate token source configuration |
| 22 | |
| 23 | Review the Kubelet configuration file, usually located at `/etc/kubernetes/kubelet/config.yaml`. Check the `authentication` and `authorization` sections to ensure all the required settings have been correctly configured. |
| 24 | |
| 25 | Make sure that the specified alternate token source is available and working correctly. |
| 26 | |
| 27 | 3. Check the API server logs |
| 28 | |
| 29 | Inspect the logs of the API server to identify any issues that may prevent the Kubelet from successfully requesting tokens. Use the following command to view the logs: |
| 30 | |
| 31 | ``` |
| 32 | kubectl logs -n kube-system kube-apiserver-<YOUR_NODE_NAME> |
| 33 | ``` |
| 34 | |
| 35 | Look for any entries related to authentication, especially if they are connected to the alternate token source. |
| 36 | |
| 37 | 4. Monitor kubelet_token_requests metric |
| 38 | |
| 39 | Keep an eye on the `kubelet_token_requests` metric using the Netdata dashboard or a monitoring system of your choice. If the number of failed requests continues to increase, this might indicate an underlying issue that requires further investigation. |
| 40 | |
| 41 | ### Useful resources |
| 42 | |
| 43 | 1. [Understanding Kubernetes authentication](https://kubernetes.io/docs/reference/access-authn-authz/authentication/) |
| 44 | 2. [Kubelet configuration reference](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/) |