016b99dc3 · cryptotaxi247/netdata-1

@@ -11,9 +11,9 @@ These components should be monitored and managed according to your organization'

11

12 ## Common Issues

13

14 -### Installation cannot finish

14 +### Timeout During Installation

15

16 -If you are getting error like:

16 +If your installation fails with this error:

17

18 ```

19 Installing netdata-cloud-onprem (or netdata-cloud-dependency) helm chart...

@@ -21,49 +21,85 @@ Installing netdata-cloud-onprem (or netdata-cloud-dependency) helm chart...

21 Error: client rate limiter Wait returned an error: Context deadline exceeded.

22 ```

23

24 -There are probably not enough resources available. Fortunately, it is very easy to verify with the `kubectl` utility. In the case of a full installation, switch the context to the cluster where On-Prem is being installed. For the Light PoC installation, SSH into the Ubuntu VM where `kubectl` is already installed and configured.

24 +This error typically indicates **insufficient cluster resources**. Here's how to diagnose and resolve the issue.

25

26 -To verify check if there are any `Pending` pods:

26 +#### Diagnosis Steps

27

28 -```shell

29 -kubectl get pods -n netdata-cloud | grep -v Running

30 -```

28 +> **Important**

29 +>

30 +> - For full installation: Ensure you're in the correct cluster context.

31 +> - For Light PoC: SSH into the Ubuntu VM with `kubectl` pre-configured.

32 +> - For Light PoC, always perform a complete uninstallation before attempting a new installation.

33

32 -To check which resource is a limiting factor pick one of the `Pending` pods and issue command:

34 +1. Check for pods stuck in Pending state:

35

34 -```shell

35 -kubectl describe pod <POD_NAME> -n netdata-cloud

36 -```

36 + ```shell

37 + kubectl get pods -n netdata-cloud | grep -v Running

38 + ```

39

38 -At the end in an `Events` section information about insufficient `CPU` or `Memory` on available nodes should appear.

39 -Please check the minimum requirements for your on-prem installation type or contact our support - `support@netdata.cloud`.

40 +2. If you find Pending pods, examine the resource constraints:

41

41 -> **Warning**

42 ->

43 -> In case of the Light PoC installations always uninstall before the next attempt.

42 + ```shell

43 + kubectl describe pod <POD_NAME> -n netdata-cloud

44 + ```

45 +

46 + Review the Events section at the bottom of the output. Look for messages about:

47 + - Insufficient CPU

48 + - Insufficient Memory

49 + - Node capacity issues

50 +

51 +3. View overall cluster resources:

52 +

53 + ```shell

54 + # Check resource allocation across nodes

55 + kubectl top nodes

56 +

57 + # View detailed node capacity

58 + kubectl describe nodes | grep -A 5 "Allocated resources"

59 + ```

60

45 -### Installation finished but login does not work

61 +#### Solution

62

47 -It depends on the installation and login type, but the underlying problem is usually located in the `values.yaml` file. In the case of Light PoC installations, this is also true, but the installation script fills in the data for the user. We can split the problem into two variants:

63 +1. Compare your available resources against the [minimum requirements](https://github.com/netdata/netdata/blob/master/docs/netdata-cloud/netdata-cloud-on-prem/installation.md#system-requirements).

64 +2. Take one of these actions:

65 + - Add more resources to your cluster.

66 + - Free up existing resources.

67

49 -1. SSO is not working - you need to check your tokens and callback URLs for a given provider. Equally important is the certificate - it needs to be trusted, and also hostname(s) under `global.public` section - make sure that FQDN is correct.

50 -2. Mail login is not working:

51 - 1. If you are using a Light PoC installation with MailCatcher, the problem usually appears if the wrong hostname was used during the installation. It needs to be a FQDN that matches the provided certificate. The usual error in such a case points to a invalid token.

52 - 2. If the magic link is not arriving for MailCatcher, it's likely because the default values were changed. In the case of using your own mail server, check the `values.yaml` file in the `global.mail.smtp` section and your network settings.

68 +### Login Issues After Installation

69

54 -If you are getting the error `Something went wrong - invalid token` and you are sure that it is not related to the hostname or the mail configuration as described above, it might be related to a dirty state of Netdata secrets. During the installation, a secret called `netdata-cloud-common` is created. By default, this secret should not be deleted by Helm and is created only if it does not exist. It stores a few strings that are mandatory for Netdata Cloud On-Prem's provisioning and continuous operation. Because they are used to hash the data in the PostgreSQL database, a mismatch will cause data corruption where the old data is not readable and the new data is hashed with the wrong string. Either a new installation is needed, or contact to our support to individually analyze the complexity of the problem.

70 +Installation may complete successfully, but login issues can occur due to configuration mismatches. This table provides a quick reference for troubleshooting common login issues after installation.

71 +

73 +|-------------------------------|---------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|

74 +| SSO Login Failure | Unable to authenticate via SSO providers | - Invalid callback URLs - Expired/invalid SSO tokens - Untrusted certificates - Incorrect FQDN in `global.public` | - Update SSO configuration in `values.yaml` - Verify certificates are valid and trusted - Ensure FQDN matches certificate |

75 +| MailCatcher Login (Light PoC) | - Magic links not arriving - "Invalid token" errors | - Incorrect hostname during installation - Modified default MailCatcher values | - Reinstall with correct FQDN - Restore default MailCatcher settings - Ensure hostname matches certificate |

76 +| Custom Mail Server Login | Magic links not arriving | - Incorrect SMTP configuration - Network connectivity issues | - Update SMTP settings in `values.yaml` - Verify network allows SMTP traffic - Check mail server logs |

77 +| Invalid Token Error | "Something went wrong - invalid token" message | - Mismatched `netdata-cloud-common` secret - Database hash mismatch - Namespace change without secret migration | - Migrate secret before namespace change - Perform fresh installation - Contact support for data recovery |

78

79 > **Warning**

80 >

58 -> If you are changing the installation namespace secret netdata-cloud-common will be created again. Make sure to transfer it beforehand or wipe postgres before new installation.

81 +> If you're modifying the installation namespace, the `netdata-cloud-common` secret will be recreated.

82 +>

83 +> **Before proceeding**: Back up the existing `netdata-cloud-common` secret. Alternatively, wipe the PostgreSQL database to prevent data conflicts.

84

85 ### Slow Chart Loading or Chart Errors

86

87 When charts take a long time to load or fail with errors, the issue typically stems from data collection challenges. The `charts` service must gather data from multiple Agents within a Room, requiring successful responses from all queried Agents.

88

65 -| -------------------- | --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

90 +|----------------------|-----------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

91 | Agent Connectivity | - Queries stall or timeout - Inconsistent chart loading | Slow Agents or unreliable network connections prevent timely data collection | Deploy additional [Parent](/docs/observability-centralization-points/README.md) nodes to provide reliable backends. The system will automatically prefer these for queries when available |

92 | Kubernetes Resources | - Service throttling - Slow data processing - Delayed dashboard updates | Resource saturation at the node level or restrictive container limits | Review and adjust container resource limits and node capacity as needed |

93 | Database Performance | - Slow query responses - Increased latency across services | PostgreSQL performance bottlenecks | Monitor and optimize database resource utilization: - CPU usage - Memory allocation - Disk I/O performance |

94 | Message Broker | - Delayed node status updates (online/offline/stale) - Slow alert transitions - Dashboard update delays | Message accumulation in Pulsar due to processing bottlenecks | - Review Pulsar configuration - Adjust microservice resource allocation - Monitor message processing rates |

95 +

96 +## Need Help?

97 +

98 +If issues persist:

99 +

100 +1. Gather the following information:

101 +

102 + - Installation logs

103 + - Your cluster specifications

104 +

105 +2. Contact support at `support@netdata.cloud`.

docs: improve on-prem troubleshooting readability (#19279)