f66b83b19 · cryptotaxi247/netdata-1

@@ -1,31 +1,71 @@

1 # Troubleshoot Agent-Cloud connectivity issues

2

3 -Learn how to troubleshoot the Netdata Agent showing as offline after claiming, so you can connect the Agent to Netdata Cloud.

3 +Learn how to troubleshoot connectivity issues leading to agents not appearing at all in Netdata Cloud, or

4 +appearing with a status other than `live`.

5

5 -When you are claiming a node, you might not be able to immediately see it online in Netdata Cloud.

6 -This could be due to an error in the claiming process or a temporary outage of some services.

6 +After installing an agent with the claiming token provided by Netdata Cloud, you should see charts from that node on

7 +Netdata Cloud within seconds. If you don't see charts, check if the node appears in the list of nodes

8 +(Nodes tab, top right Node filter, or Manage Nodes screen). If your node does not appear in the list, or it does appear with a status other than "Live", this guide will help you troubleshoot what's happening.

9

8 -We identified some scenarios that might cause this delay and possible actions you could take to overcome each situation.

10 + The most common explanation for connectivity issues usually falls into one of the following three categories:

11

10 -The most common explanation for the delay usually falls into one of the following three categories:

12 +- If the node does not appear at all in Netdata Cloud, [the claiming process was unsuccessful](#the-claiming-process-was-unsuccessful).

13 +- If the node appears as in Netdata Cloud, but is in the "Unseen" state, [the Agent was claimed but can not connect](#the-agent-was-claimed-but-can-not-connect).

14 +- If the node appears as in Netdata Cloud as "Offline" or "Stale", it is a [previously connected agent that can no longer connect](#previously-connected-agent-that-can-no-longer-connect).

15

12 -- [Troubleshoot Agent-Cloud connectivity issues](#troubleshoot-agent-cloud-connectivity-issues)

13 - - [The claiming process of the kickstart script was unsuccessful](#the-claiming-process-of-the-kickstart-script-was-unsuccessful)

14 - - [The kickstart script auto-claimed the Agent but there was no error message displayed](#the-kickstart-script-auto-claimed-the-agent-but-there-was-no-error-message-displayed)

15 - - [Claiming on an older, deprecated version of the Agent](#claiming-on-an-older-deprecated-version-of-the-agent)

16 - - [Network issues while connecting to the Cloud](#network-issues-while-connecting-to-the-cloud)

17 - - [Verify that your IP is whitelisted from Netdata Cloud](#verify-that-your-ip-is-whitelisted-from-netdata-cloud)

18 - - [Make sure that your node has internet connectivity and can resolve network domains](#make-sure-that-your-node-has-internet-connectivity-and-can-resolve-network-domains)

16 +## The claiming process was unsuccessful

17

20 -## The claiming process of the kickstart script was unsuccessful

18 +If the claiming process fails, the node will not appear at all in Netdata Cloud.

19

22 -Here, we will try to define some edge cases you might encounter when claiming a node.

20 +First ensure that you:

21 +- Use the newest possible stable or nightly version of the agent (at least v1.32).

22 +- Your node can successfully issue an HTTPS request to https://api.netdata.cloud

23

24 -### The kickstart script auto-claimed the Agent but there was no error message displayed

24 +Other possible causes differ between kickstart installations and Docker installations.

25

26 -The kickstart script will install/update your Agent and then try to claim the node to the Cloud (if tokens are provided). To

27 -complete the second part, the Agent must be running. In some platforms, the Netdata service cannot be enabled by default

28 -and you must do it manually, using the following steps:

26 +### Verify your node can access Netdata Cloud

27 +

28 +If you run either `curl` or `wget` to do an HTTPS request to https://api.netdata.cloud, you should get

29 +back a 404 response. If you do not, check your network connectivity, domain resolution,

30 +and firewall settings for outbound connections.

31 +

32 +If your firewall is configured to completely prevent outbound connections, you need to whitelist `api.netdata.cloud` and `mqtt.netdata.cloud`. If you can't whitelist domains in your firewall, you can whitelist the IPs that the hostnames resolve to, but keep in mind that they can change without any notice.

33 +

34 +If you use an outbound proxy, you need to [take some extra steps]( https://github.com/netdata/netdata/blob/master/claim/README.md#connect-through-a-proxy).

35 +

36 +### Troubleshoot claiming with kickstart.sh

37 +

38 +Claiming is done by executing `netdata-claim.sh`, a script that is usually located under `${INSTALL_PREFIX}/netdata/usr/sbin/netdata-claim.sh`. Possible error conditions we have identified are:

39 +- No script found at all in any of our search paths.

40 +- The path where the claiming script should be does not exist.

41 +- The path exists, but is not a file.

42 +- The path is a file, but is not executable.

43 +Check the output of the kickstart script for any reported errors claiming and verify that the claiming script exists

44 +and can be executed.

45 +

46 +### Troubleshoot claiming with Docker

47 +

48 +First verify that the NETDATA_CLAIM_TOKEN parameter is correctly configured and then check for any errors during

49 +initialization of the container.

50 +

51 +The most common issue we have seen claiming nodes in Docker is [running on older hosts with seccomp enabled](https://github.com/netdata/netdata/blob/master/claim/README.md#known-issues-on-older-hosts-with-seccomp-enabled).

52 +

53 +## The Agent was claimed but can not connect

54 +

55 +Agents that appear on the cloud with state "Unseen" have successfully been claimed, but have never

56 +been able to successfully establish an ACLK connection.

57 +

58 +Agents that appear with state "Offline" or "Stale" were able to connect at some point, but are currently not

59 +connected. The difference between the two is that "Stale" nodes had some of their data replicated to a

60 +parent node that is still connected.

61 +

62 +### Verify that the agent is running

63 +

64 +#### Troubleshoot connection establishment with kickstart.sh

65 +

66 +The kickstart script will install/update your Agent and then try to claim the node to the Cloud

67 +(if tokens are provided). To complete the second part, the Agent must be running. In some platforms,

68 +the Netdata service cannot be enabled by default and you must do it manually, using the following steps:

69

70 1. Check if the Agent is running:

71

@@ -53,17 +93,39 @@ and you must do it manually, using the following steps:

93 > In some cases a simple restart of the Agent can fix the issue.

94 > Read more about [Starting, Stopping and Restarting the Agent](https://github.com/netdata/netdata/blob/master/docs/configure/start-stop-restart.md).

95

56 -## Claiming on an older, deprecated version of the Agent

96 +#### Troubleshoot connection establishment with Docker

97 +

98 +If a Netdata container exits or is killed before it properly starts, it may be able to complete the claiming

99 +process, but not have enough time to establish the ACLK connection.

100 +

101 +### Verify that your firewall allows websockets

102 +

103 +The agent initiates an SSL connection to `api.netdata.cloud` and then upgrades that connection to use secure

104 +websockets. Some firewalls completely prevent the use of websockets, even for outbound connections.

105 +

106 +## Previously connected agent that can no longer connect

107

58 -Make sure that you are using the latest version of Netdata if you are using the [Claiming script](https://github.com/netdata/netdata/blob/master/claim/README.md#claiming-script).

108 +The states "Offline" and "Stale" suggest that the agent was able to connect at some point in the past, but

109 +that it is currently not connected.

110

60 -With the introduction of our new architecture, Agents running versions lower than `v1.32.0` can face claiming problems, so we recommend you [update the Netdata Agent](https://github.com/netdata/netdata/blob/master/packaging/installer/UPDATE.md) to the latest stable version.

111 +### Verify that network connectivity is still possible

112

62 -## Network issues while connecting to the Cloud

113 +Verify that you can still issue HTTPS requests to api.netdata.cloud and that no firewall or proxy changes were made.

114

64 -### Verify that your IP is whitelisted from Netdata Cloud

115 +### Verify that the claiming info is persisted

116

66 -Most of the nodes change IPs dynamically. It is possible that your current IP has been restricted from accessing `api.netdata.cloud` due to security concerns.

117 +If you use Docker, verify that the contents of `/var/lib/netdata` are preserved across container restarts, using a persistent volume.

118 +

119 +### Verify that the claiming info is not cloned

120 +

121 +A relatively common case we have seen especially with VMs is two or more nodes sharing the same credentials.

122 +This happens if you claim a node in a VM and then create an image based on that node. Netdata can't properly

123 +work this way, as we have unique node identification information under `/var/lib/netdata`.

124 +

125 +### Verify that your IP is not blocked by Netdata Cloud

126 +

127 +Most of the nodes change IPs dynamically. It is possible that your current IP has been restricted from accessing `api.netdata.cloud` due to security concerns, usually because it was spamming Netdata Coud with too many

128 +failed requests (old versions of the agent).

129

130 To verify this:

131

@@ -83,31 +145,3 @@ To verify this:

145

146 - Contact our team to whitelist your IP by submitting a ticket in the [Netdata forum](https://community.netdata.cloud/)

147 - Change your node's IP

86 -

87 -### Make sure that your node has internet connectivity and can resolve network domains

88 -

89 -1. Try to reach a well known host:

90 -

91 - ```bash

92 - ping 8.8.8.8

93 - ```

94 -

95 -2. If you can reach external IPs, then check your domain resolution.

96 -

97 - ```bash

98 - host api.netdata.cloud

99 - ```

100 -

101 - The expected output should be something like this:

102 -

103 - ```bash

104 - api.netdata.cloud is an alias for main-ingress-545609a41fcaf5d6.elb.us-east-1.amazonaws.com.

105 - main-ingress-545609a41fcaf5d6.elb.us-east-1.amazonaws.com has address 54.198.178.11

106 - main-ingress-545609a41fcaf5d6.elb.us-east-1.amazonaws.com has address 44.207.131.212

107 - main-ingress-545609a41fcaf5d6.elb.us-east-1.amazonaws.com has address 44.196.50.41

108 - ```

109 -

110 - > ### Info

111 - >

112 - > There will be cases in which the firewall restricts network access. In those cases, you need to whitelist `api.netdata.cloud` and `mqtt.netdata.cloud` domains to be able to see your nodes in Netdata Cloud.

113 - > If you can't whitelist domains in your firewall, you can whitelist the IPs that the above command will produce, but keep in mind that they can change without any notice.

Update troubleshooting-agent-with-cloud-connection.md (#15029)