Docs: Combined claiming+ACLK documentation (#8724)
* Init new documents * Finalize draft of combined claiming doc * Add notice to anonymous stats * Remove .. from links * Update none proxy setting * Changes for Andrew and Manos * Remove E2EE from ACLK * Add details about netdata user
Joel Hans committed
Apr 16, 2020 at 13:57 UTC
3bac7654b70d1a61416f2c13d8ece679b20f7e5a
3 files changed
+254
-51
aclk/README.md
+33
-10
@@ -1,25 +1,48 @@
1
<!--
2
---
3
-title: "Agent-cloud link (ACLK)"
3
+title: "Agent-Cloud link (ACLK)"
4
+description: "The Agent-Cloud link (ACLK) is the mechanism responsible for connecting a Netdata agent to Netdata Cloud."
5
+date: 2020-04-15
6
custom_edit_url: https://github.com/netdata/netdata/edit/master/aclk/README.md
7
---
8
-->
9
10
# Agent-cloud link (ACLK)
11
12
+The Agent-Cloud link (ACLK) is the mechanism responsible for connecting a Netdata Agent to Netdata Cloud. The ACLK uses
13
+[MQTT](https://en.wikipedia.org/wiki/MQTT) over secure websockets to first create, persist, encrypt the connection, and
14
+then enable the features found in Netdata Cloud. _No data is exchanged with Netdata Cloud until you claim a node._
15
11
-## Configuration Options
16
+Read our [claiming documentation](/claim/README.md) for a guide for claiming a node using the ACLK and additional
17
+troubleshooting and reference information.
18
13
-In `netdata.conf`:
19
+## Enable and configure the ACLK
20
15
-```ini
21
+The ACLK is enabled by default and automatically configured if the prerequisites installed correctly. You can see this
22
+in the `[cloud]` section of `netdata.conf`.
23
+
24
+```conf
25
[cloud]
17
- proxy = none
26
+ cloud base url = https://app.netdata.cloud
27
```
28
20
-Parameter proxy can take one of the following values:
29
+If your Agent needs to use a proxy to access the internet, you must [set up a proxy for
30
+claiming](/claim/README.md#claiming-through-a-proxy).
31
+
32
+## Disable the ACLK
33
+
34
+You have two options if you prefer to disable the ACLK and not use Netdata Cloud:
35
+
36
+1. Pass `--disable-cloud` to `netdata-installer.sh` during installation. When you pass this parameter, the installer
37
+ does not download or compile any extra libraries, and the Agent behaves as though the ACLK, and thus Netdata Cloud,
38
+ does not exist. ACLK functionality is available in the Agent but remains fully inactive.
39
+
40
+2. Change a runtime setting in your `netdata.conf` file. This setting only stops the Agent from attempting any
41
+ connection via the ACLK, but does not prevent the installer from downloading and compiling the ACLK's dependencies.
42
+
43
+```conf
44
+[global]
45
+ netdata cloud = disable
46
+```
47
22
-- `env` - the default (try to read environment variables `http_proxy` and `socks_proxy`)
23
-- `none` - do not use any proxy (even if system configured otherwise)
24
-- `socks5[h]://[user:pass@]host:ip` - will use specified socks proxy
25
-- `http://[user:pass@]host:ip` - will use specified http proxy
48
+[](<>)
claim/README.md
+214
-41
@@ -1,41 +1,235 @@
1
<!--
2
---
3
title: "Agent claiming"
4
+description: "Agent claiming allows a Netdata Agent, running on a distributed node, to securely connect to Netdata Cloud. A Space's administrator creates a claiming token, which is used to add an Agent to their Space via the Agent-Cloud link."
5
+date: 2020-04-15
6
custom_edit_url: https://github.com/netdata/netdata/edit/master/claim/README.md
7
---
8
-->
9
10
# Agent claiming
11
10
-Agent claiming is part of the onboarding process when creating a workspace in Netdata Cloud. Each workspace gets its own
11
-common invitation mechanism, which begins with the administrators of the workspace creating a **claiming-token**. They,
12
-or other users is their organization, can then use the claiming-token to add an agent to their workspace.
12
+Agent claiming allows a Netdata Agent, running on a distributed node, to securely connect to Netdata Cloud. A Space's
13
+administrator creates a **claiming token**, which is used to add an Agent to their Space via the [Agent-Cloud link
14
+(ACLK)](/aclk/README.md).
15
14
-To claim a Netdata agent, you first send a claiming request to Netdata Cloud (from the agent node). Once the
15
-Netdata Cloud validates the claiming request of the agent (based on the claiming token), and returns a successful
16
-result, the node is considered claimed.
16
+Claiming nodes is a security feature in Netdata Cloud. Through the process of claiming, you demonstrate in a few ways
17
+that you have administrative access to that node and the configuration settings for its Agent. By logging into the node,
18
+you prove you have access, and by using the claiming script or the Netdata command line, you prove you have write access
19
+and administrative privileges.
20
18
-## Claiming script
21
+Only the administrators of a Space in Netdata Cloud can view the claiming token and accompanying script generated by
22
+Netdata Cloud.
23
20
-The user can claim an agent by directly calling the `netdata-claim.sh` script **as the netdata user** and passing the
21
-following arguments:
24
+> The claiming process ensures no third party can add your node, and then view your node's metrics, in a Cloud account,
25
+> Space, or War Room that you did not authorize.
26
+
27
+By claiming a node, you opt-in to sending data from your Agent to Netdata Cloud via the ACLK. This data is encrypted by
28
+TLS while it is in transit. We use the the RSA keypair created during claiming to authenticate the identity of the agent
29
+when it connects to the Cloud. While the data does flow through Netdata Cloud servers on its way from Agents to the
30
+browser, we do not store or log it.
31
+
32
+## How to claim a node
33
+
34
+You can claim a node during the Cloud onboarding process, or after you created a Space by clicking on the **USER's
35
+Space** dropdown, then **Manage Claimed Nodes**.
36
+
37
+> Only the administrators of a Space in Netdata Cloud can view the claiming token and accompanying script generated by
38
+> Netdata Cloud.
39
+
40
+To claim a node, copy the script given by Cloud. **You must run this script as the user running the** `netdata`
41
+**service**, which is usually the `netdata` user. You have two options: Switch to the appropriate user yourself, or run
42
+the command using `sudo` to automatically manage permissions.
43
+
44
+By switching to the `netdata` user:
45
+
46
+```bash
47
+sudo su -s /bin/bash netdata
48
+netdata-claim.sh -token=TOKEN -rooms=ROOM1,ROOM2 -url=https://app.netdata.cloud
49
+```
50
+
51
+With `sudo`:
52
+
53
+```bash
54
+sudo netdata-claim.sh -token=TOKEN -rooms=ROOM1,ROOM2 -url=https://app.netdata.cloud
55
+```
56
+
57
+Hit **Enter**. The script should return `Agent was successfully claimed.`.
58
+
59
+> Your node may need up to 60 seconds to connect to Netdata Cloud after finishing the claiming process. Please be
60
+> patient!
61
+
62
+If the claiming script returns errors, see the [troubleshooting information](#troubleshooting).
63
+
64
+### Claiming through a proxy
65
+
66
+A Space's administrator can claim a node through a SOCKS5 or HTTP(S) proxy.
67
+
68
+You should first configure the proxy in the `[cloud]` section of `netdata.conf`. The proxy settings you specify here
69
+will also be used to tunnel the ACLK. The default `proxy` setting is `none`.
70
+
71
+```ini
72
+[cloud]
73
+ proxy = none
74
+```
75
+
76
+The `proxy` setting can take one of the following values:
77
+
78
+- `none`: Do not use a proxy, even if the system configured otherwise.
79
+- `env`: Try to read proxy settings from set environment variables `http_proxy`/`socks_proxy`.
80
+- `socks5[h]://[user:pass@]host:ip`: The ACLK and claiming will use the specified SOCKS5 proxy.
81
+- `http://[user:pass@]host:ip`: The ACLK and claiming will use the specified HTTP(S) proxy.
82
+
83
+For example, a SOCKS5 proxy setting may look like the following:
84
+
85
+```conf
86
+[cloud]
87
+ proxy = socks5h://203.0.113.0:1080 # With an IP address
88
+ proxy = socks5h://proxy.example.com:1080 # With a URL
89
+```
90
+
91
+You can now move on to claiming. Be sure to switch to the `netdata` user or use `sudo` as explained in the [step above](#how-to-claim-a-node).
92
+
93
+When you claim with the `netdata-claim.sh` script, add the `-proxy=` parameter and append the same proxy setting you
94
+added to `netdata.conf`.
95
+
96
+```bash
97
+netdata-claim.sh -token=MYTOKEN1234567 -rooms=room1,room2 -url=https://app.netdata.cloud -proxy=socks5h://203.0.113.0:1080
98
+```
99
+
100
+Hit **Enter**. The script should return `Agent was successfully claimed.`.
101
+
102
+> Your node may need up to 60 seconds to connect to Netdata Cloud after finishing the claiming process. Please be
103
+> patient!
104
+
105
+If the claiming script returns errors, see the [troubleshooting information](#troubleshooting).
106
+
107
+### Troubleshooting
108
+
109
+If you're having trouble claiming a node, this may be because the ACLK cannot connect to Cloud.
110
+
111
+With the Netdata Agent running, visit `http://127.0.0.1/api/v1/info` in your browser. The returned JSON contains four
112
+keys that will be helpful to diagnose any issues you might be having with the ACLK or claiming process.
113
+
114
+```json
115
+ "cloud-enabled"
116
+ "cloud-available"
117
+ "agent-claimed"
118
+ "aclk-available"
119
+```
120
+
121
+Use these keys and the information below to troubleshoot the ACLK.
122
+
123
+#### cloud-enabled is false
124
+
125
+If `cloud-enabled` is `false`, you probably ran the installer with `--disable-cloud` option.
126
+
127
+Additionally, check that the `netdata cloud` setting in `netdata.conf` is set to `enable`:
128
+
129
+```ini
130
+[general]
131
+ netadata cloud = enable
132
+```
133
+
134
+To fix this issue, reinstall Netdata using your [preferred method](/packaging/installer/README.md) and do not add the
135
+`--disable-cloud` option.
136
+
137
+#### cloud-available is false
138
+
139
+If `cloud-available` is `false` after you verified Cloud is enabled in the previous step, the most likely issue is that
140
+Cloud features failed to build during installation.
141
+
142
+If Cloud features fail to build, the installer continues and finishes the process without Cloud functionality as opposed
143
+to failing the installation altogether. We do this to ensure the Agent will always finish installing.
144
+
145
+If you can't see an explicit error in the installer's output, you can run the installer with the `--require-cloud`
146
+option. This option causes the installation to fail if Cloud functionality can't be built and enabled, and the
147
+installer's output should give you more error details.
148
+
149
+You may see one of the following error messages during installation:
150
+
151
+- Failed to build libmosquitto. The install process will continue, but you will not be able to connect this node to
152
+ Netdata Cloud.
153
+- Unable to fetch sources for libmosquitto. The install process will continue, but you will not be able to connect
154
+ this node to Netdata Cloud.
155
+- Failed to build libwebsockets. The install process will continue, but you may not be able to connect this node to
156
+ Netdata Cloud.
157
+- Unable to fetch sources for libwebsockets. The install process will continue, but you may not be able to connect
158
+ this node to Netdata Cloud.
159
+
160
+One common cause of the installer failing to build Cloud features is not having one of the following dependencies on
161
+your system: `cmake` and OpenSSL, including the `devel` package.
162
+
163
+You can also look for error messages in `/var/log/netdata/error.log`. Try one of the following two commands to search
164
+for ACLK-related errors.
165
+
166
+```bash
167
+less /var/log/netdata/error.log
168
+grep -i ACLK /var/log/netdata/error.log
169
+```
170
+
171
+If the installer's output does not help you enable Cloud features, contact us by [creating an issue on
172
+GitHub](https://github.com/netdata/netdata/issues/new?labels=bug%2C+needs+triage%2C+ACLK&template=bug_report.md&title=The+installer+failed+to+prepare+the+required+dependencies+for+Netdata+Cloud+functionality)
173
+with details about your system and relevant output from `error.log`.
174
+
175
+#### agent-claimed is false
176
+
177
+You must [claim your node](#how-to-claim-a-node).
178
+
179
+#### aclk-available is false
180
+
181
+If `aclk-available` is `false` and all other keys are `true`, your Agent is having trouble connection to the Cloud
182
+through the ACLK. Please check your system's firewall.
183
+
184
+If your Agent needs to use a proxy to access the internet, you must [set up a proxy for
185
+claiming](#claiming-through-a-proxy).
186
+
187
+If you are certain firewall and proxy settings are not the issue, you should consult the Agent's `error.log` at
188
+`/var/log/netdata/error.log` and contact us by [creating an issue on
189
+GitHub](https://github.com/netdata/netdata/issues/new?labels=bug%2C+needs+triage%2C+ACLK&template=bug_report.md&title=ACLK-available-is-false)
190
+with details about your system and relevant output from `error.log`.
191
+
192
+### Unclaim (remove) an Agent from Netdata Cloud
193
+
194
+The best method to remove an Agent from Netdata Cloud is to unclaim it by deleting the `claim.d/` directory in your
195
+Netdata configuration directory.
196
+
197
+```bash
198
+cd /etc/netdata # Replace with your Netdata configuration directory, if not /etc/netdata/
199
+rm -rf claim.d/
200
+```
201
+
202
+> You may need to use `sudo` or another method of elevating your privileges.
203
+
204
+Once you delete the `claim.d/` directory, the ACLK will not connect to Cloud the next time the Agent starts, and Cloud
205
+will then remove it from the interface.
206
+
207
+## Claiming reference
208
+
209
+In the sections below, you can find reference material for the claiming script, claiming via the Agent's command line
210
+tool, and details about the files found in `claim.d`.
211
+
212
+### Claiming script
213
+
214
+A Space's administrator can claim an Agent by directly calling the `netdata-claim.sh` script **as the `netdata` user**
215
+and passing the following arguments:
216
217
```sh
218
-token=TOKEN
25
- where TOKEN is the workspace claiming-token.
219
+ where TOKEN is the Space's claiming token.
220
-rooms=ROOM1,ROOM2,...
27
- where ROOMX is the workspace war-room to join. This list is optional.
221
+ where ROOMX is the War Room this node should be added to. This list is optional.
222
-url=URL_BASE
223
where URL_BASE is the Netdata Cloud endpoint base URL. By default, this is https://netdata.cloud.
224
-id=AGENT_ID
31
- where AGENT_ID is the unique identifier of the agent. This is the agent's MACHINE_GUID by default.
225
+ where AGENT_ID is the unique identifier of the Agent. This is the Agent's MACHINE_GUID by default.
226
-hostname=HOSTNAME
227
where HOSTNAME is the result of the hostname command by default.
228
-proxy=PROXY_URL
229
where PROXY_URL is the endpoint of a SOCKS5 proxy.
230
```
231
38
-For example, the following command claims an agent and adds it to rooms `room1` and `room2`:
232
+For example, the following command claims an Agent and adds it to rooms `room1` and `room2`:
233
234
```sh
235
netdata-claim.sh -token=MYTOKEN1234567 -rooms=room1,room2
@@ -47,11 +241,12 @@ You should then update the `netdata` service about the result with `netdatacli`:
241
netdatacli reload-claiming-state
242
```
243
50
-This reloads the agent claiming state from disk.
244
+This reloads the Agent claiming state from disk.
245
52
-## Netdata agent command line
246
+### Netdata Agent command line
247
54
-The user can trigger agent claiming by calling the `netdata` service binary with the additional command line parameters:
248
+If a Netdata Agent is running, the Space's administrator can claim a node using the `netdata` service binary with
249
+additional command line parameters:
250
251
```sh
252
-W "claim -token=TOKEN -rooms=ROOM1,ROOM2"
@@ -63,10 +258,10 @@ For example:
258
/usr/sbin/netdata -D -W "claim -token=MYTOKEN1234567 -rooms=room1,room2"
259
```
260
66
-If need be, the user can override the agent's defaults by providing additional arguments like those described
261
+If need be, the user can override the Agent's defaults by providing additional arguments like those described
262
[here](#claiming-script).
263
69
-## Claiming directory
264
+### Claiming directory
265
266
Netdata stores the agent claiming-related state in the user configuration directory under `claim.d`, e.g. in
267
`/etc/netdata/claim.d`. The user can put files in this directory to provide defaults to the `-token` and `-rooms`
@@ -75,29 +270,7 @@ arguments. These files should be owned **by the `netdata` user**.
270
The `claim.d/token` file should contain the claiming-token and the `claim.d/rooms` file should contain the list of
271
war-rooms.
272
78
-The user can also put the Cloud endpoint's full certificate chain in `claim.d/cloud_fullchain.pem` so that the agent
273
+The user can also put the Cloud endpoint's full certificate chain in `claim.d/cloud_fullchain.pem` so that the Agent
274
can trust the endpoint if necessary.
275
81
-## Using a proxy
82
-
83
-Claiming can be performed through a SOCKS5 proxy. To do this when calling the script directly supply the proxy
84
-endpoint as:
85
-
86
-```
87
-netdata-claim.sh -token=MYTOKEN1234567 -rooms=room1,room2 -proxy=socks5h://127.0.0.1:11081
88
-```
89
-
90
-When claiming via the `netdata` binary set the following options in the config:
91
-```
92
-[cloud]
93
- proxy = socks5://X.X.X.X:YYYY
94
-```
95
-Proceed to claim using the command-line syntax:
96
-```
97
-/usr/sbin/netdata -D -W "claim -token=MYTOKEN1234567 -rooms=room1,room2"
98
-```
99
-
100
-Please note - if you supply the proxy endpoint in the configuration then it will also be used to tunnel
101
-the agent cloud link as well.
102
-
276
[](<>)
docs/anonymous-statistics.md
+7
@@ -70,6 +70,13 @@ events is:
70
Furthermore, the FATAL event sends the Netdata process & thread name, along with the source code function, source code
71
filename and source code line number of the fatal error.
72
73
+Starting with v1.21, we additionally collect information about:
74
+
75
+- Failures to build the dependencies required to use Cloud features.
76
+- Unavailability of Cloud features in an agent.
77
+- Failures to connect to the Cloud in case the agent has been [claimed](/claim/README.md). This includes error codes
78
+ to inform the Netdata team about the reason why the connection failed.
79
+
80
To see exactly what and how is collected, you can review the script template `daemon/anonymous-statistics.sh.in`. The
81
template is converted to a bash script called `anonymous-statistics.sh`, installed under the Netdata `plugins
82
directory`, which is usually `/usr/libexec/netdata/plugins.d`.