Overhaul streaming documentation (#10709)
* Init quickstart * Switch to dbengine * Init new files * Init new files * Finish with reference * Add descriptions * Restore existing /streaming/README.md to original * Reconfigure settings per Thiago's feedback * Fixes for Vlad * Fix typo * Add optional TLS/SSL section * Add link to SSL options
Joel Hans committed
Apr 29, 2021 at 07:59 UTC
db60f4928d55628d8a88b0fcf7bcaeb8d02069ce
4 files changed
+739
-7
docs/metrics-storage-management/enable-streaming.mdx
new
+151
@@ -0,0 +1,151 @@
1
+---
2
+title: "Enable streaming between nodes"
3
+description: "With metrics streaming enabled, you can not only replicate metrics data into a second database, but also view dashboards and trigger alarm notifications for multiple nodes in parallel."
4
+type: how-to
5
+custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/metrics-storage-management/enable-streaming.mdx
6
+---
7
+
8
+# Enable streaming between nodes
9
+
10
+The simplest streaming configuration is **replication**, in which a child node streams its metrics in real time to a
11
+parent node, and both nodes retain metrics in their own databases.
12
+
13
+To configure replication, you need two nodes, each running Netdata. First you'll first enable streaming on your parent
14
+node, then enable streaming on your child node. When you're finished, you'll be able to see the child node's metrics in
15
+the parent node's dashboard, quickly switch between the two dashboards, and be able to serve [alarm
16
+notifications](/docs/monitor/enable-notifications.md) from either or both nodes.
17
+
18
+## Enable streaming on the parent node
19
+
20
+First, log onto the node that will act as the parent.
21
+
22
+Run `uuidgen` to create a new API key, which is a randomly-generated machine GUID the Netdata Agent uses to identify
23
+itself while initiating a streaming connection. Copy that into a separate text file for later use.
24
+
25
+> Find out how to [install `uuidgen`](https://command-not-found.com/uuidgen) on your node if you don't already have it.
26
+
27
+Next, open `stream.conf` using [`edit-config`](/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files)
28
+from within the [Netdata config directory](/docs/configure/nodes.md#the-netdata-config-directory).
29
+
30
+```bash
31
+cd /etc/netdata
32
+sudo ./edit-config stream.conf
33
+```
34
+
35
+Scroll down to the section beginning with `[API_KEY]`. Paste the API key you generated earlier between the brackets, so
36
+that it looks like the following:
37
+
38
+```conf
39
+[11111111-2222-3333-4444-555555555555]
40
+```
41
+
42
+Set `enabled` to `yes`, and `default memory mode` to `dbengine`. Leave all the other settings as their defaults. A
43
+simplified version of the configuration, minus the commented lines, looks like the following:
44
+
45
+```conf
46
+[11111111-2222-3333-4444-555555555555]
47
+ enabled = yes
48
+ default memory mode = dbengine
49
+```
50
+
51
+Save the file and close it, then restart Netdata with `sudo systemctl restart netdata`, or the [appropriate
52
+method](/docs/configure/start-stop-restart.md) for your system.
53
+
54
+## Enable streaming on the child node
55
+
56
+Connect to your child node with SSH.
57
+
58
+Open `stream.conf` again. Scroll down to the `[stream]` section and set `enabled` to `yes`. Paste the IP address of your
59
+parent node at the end of the `destination` line, and paste the API key generated on the parent node onto the `api key`
60
+line.
61
+
62
+Leave all the other settings as their defaults. A simplified version of the configuration, minus the commented lines,
63
+looks like the following:
64
+
65
+```conf
66
+[stream]
67
+ enabled = yes
68
+ destination = 203.0.113.0
69
+ api key = 11111111-2222-3333-4444-555555555555
70
+```
71
+
72
+Save the file and close it, then restart Netdata with `sudo systemctl restart netdata`, or the [appropriate
73
+method](/docs/configure/start-stop-restart.md) for your system.
74
+
75
+## Enable TLS/SSL on streaming (optional)
76
+
77
+While encrypting the connection between your parent and child nodes is recommended for security, it's not required to
78
+get started. If you're not interested in encryption, skip ahead to [view streamed
79
+metrics](#view-streamed-metrics-in-netdata-s-dashboard).
80
+
81
+In this example, we'll use self-signed certificates.
82
+
83
+On the **parent** node, use OpenSSL to create the key and certificate, then use `chown` to make the new files readable
84
+by the `netdata` user.
85
+
86
+```bash
87
+sudo openssl req -newkey rsa:2048 -nodes -sha512 -x509 -days 365 -keyout /etc/netdata/ssl/key.pem -out /etc/netdata/ssl/cert.pem
88
+sudo chown netdata:netdata /etc/netdata/ssl/cert.pem k/etc/netdata/ssl/ey.pem
89
+```
90
+
91
+Next, enforce TLS/SSL on the web server. Open `netdata.conf`, scroll down to the `[web]` section, and look for the `bind
92
+to` setting. Add `^SSL=force` to turn on TLS/SSL. See the [web server
93
+reference](/web/server/README.md#enabling-tls-support) for other TLS/SSL options.
94
+
95
+```conf
96
+[web]
97
+ bind to = *=dashboard|registry|badges|management|streaming|netdata.conf^SSL=force
98
+```
99
+
100
+Next, connect to the **child** node and open `stream.conf`. Add `:SSL` to the end of the existing `destination` setting
101
+to connect to the parent using TLS/SSL. Uncomment the `ssl skip certificate verification` line to allow the use of
102
+self-signed certificates.
103
+
104
+```conf
105
+[stream]
106
+ enabled = yes
107
+ destination = 203.0.113.0:SSL
108
+ ssl skip certificate verification = yes
109
+ api key = 11111111-2222-3333-4444-555555555555
110
+```
111
+
112
+Restart both the parent and child nodes with `sudo systemctl restart netdata`, or the [appropriate
113
+method](/docs/configure/start-stop-restart.md) for your system, to stream encrypted metrics using TLS/SSL.
114
+
115
+## View streamed metrics in Netdata's dashboard
116
+
117
+At this point, the child node is streaming its metrics in real time to its parent. Open the local Agent dashboard for
118
+the parent by navigating to `http://PARENT-NODE:19999` in your browser, replacing `PARENT-NODE` with its IP address or
119
+hostname.
120
+
121
+This dashboard shows parent metrics. To see child metrics, open the left-hand sidebar with the hamburger icon
122
+
123
+in the top panel. Both nodes appear under the **Replicated Nodes** menu. Click on either of the links to switch between
124
+separate parent and child dashboards.
125
+
126
+
128
+
129
+The child dashboard is also available directly at `http://PARENT-NODE:19999/host/CHILD-HOSTNAME`, which in this example
130
+is `http://203.0.113.0:19999/host/netdata-child`.
131
+
132
+## What's next?
133
+
134
+Now that you have a basic streaming setup with replication, you may want to tweak the configuration to eliminate the
135
+child database, disable the child dashboard, or enable SSL on the streaming connection between the parent and child.
136
+
137
+See the [streaming reference
138
+doc](/docs/metrics-storage-management/reference-streaming.mdx#supported-streaming-configurations) for details about
139
+other possible configurations.
140
+
141
+When using Netdata's default TSDB (`dbengine`), the parent node maintains separate, parallel databases for itself and
142
+every child node streaming to it. Each instance is sized identically based on the `dbengine multihost disk space`
143
+setting in `netdata.conf`. See our doc on [changing metrics retention](/docs/store/change-metrics-storage.md) for
144
+details.
145
+
146
+### Related information & further reading
147
+
148
+- Streaming
149
+ - [How Netdata streams metrics](/docs/metrics-storage-management/how-streaming-works.mdx)
150
+ - **[Enable streaming between nodes](/docs/metrics-storage-management/enable-streaming.mdx)**
151
+ - [Streaming reference](/docs/metrics-storage-management/reference-streaming.mdx)
\ No newline at end of file
docs/metrics-storage-management/how-streaming-works.mdx
new
+92
@@ -0,0 +1,92 @@
1
+---
2
+title: "How metrics streaming works"
3
+description: "Netdata's real-time streaming allows you to replicate metrics data across multiple nodes, or centralize all your metrics data into a single time-series database (TSDB)."
4
+type: explanation
5
+custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/metrics-storage-management/how-streaming-works.mdx
6
+---
7
+
8
+# How metrics streaming works
9
+
10
+Each node running Netdata can stream the metrics it collects, in real time, to another node. Streaming allows you to
11
+replicate metrics data across multiple nodes, or centralize all your metrics data into a single time-series database
12
+(TSDB).
13
+
14
+When one node streams metrics to another, the node receiving metrics can visualize them on the
15
+[dashboard](/docs/visualize/interact-dashboards-charts.md), run health checks to [trigger
16
+alarms](/docs/monitor/view-active-alarms.md) and [send notifications](/docs/monitor/enable-notifications.md), and
17
+[export](/docs/export/external-databases.md) all metrics to an external TSDB. When Netdata streams metrics to another
18
+Netdata, the receiving one is able to perform everything a Netdata instance is capable of.
19
+
20
+Streaming lets you decide exactly how you want to store and maintain metrics data. While we believe Netdata's
21
+[distributed architecture](/docs/store/distributed-data-architecture.md) is ideal for speed and scale, streaming
22
+provides centralization options for those who want to maintain only a single TSDB instance.
23
+
24
+## Streaming basics
25
+
26
+There are three types of nodes in Netdata's streaming ecosystem.
27
+
28
+- **Parent**: A node, running Netdata, that receives streamed metric data.
29
+- **Child**: A node, running Netdata, that streams metric data to one or more parent.
30
+- **Proxy**: A node, running Netdata, that receives metric data from a child and "forwards" them on to a
31
+ separate parent node.
32
+
33
+Netdata uses API keys, which are just random GUIDs, to authorize the communication between child and parent nodes. We
34
+recommend using `uuidgen` for generating API keys, which can then be used across any number of streaming connections.
35
+Or, you can generate unique API keys for each parent-child relationship.
36
+
37
+Once the parent node authorizes the child's API key, the child can start streaming metrics.
38
+
39
+It's important to note that the streaming connection uses TCP, UDP, or Unix sockets, _not HTTP_. To proxy streaming
40
+metrics, you need to use a proxy that tunnels [OSI layer 4-7
41
+traffic](https://en.wikipedia.org/wiki/OSI_model#Layer_4:_Transport_Layer) without interfering with it, such as
42
+[SOCKS](https://en.wikipedia.org/wiki/SOCKS) or Nginx's [TCP/UDP load
43
+balancing](https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-udp-load-balancer/).
44
+
45
+## Supported streaming configurations
46
+
47
+Netdata supports any combination of parent, child, and proxy nodes that you can imagine. Any node can act as both a
48
+parent, child, or proxy at the same time, sending or receiving streaming metrics from any number of other nodes.
49
+
50
+Here are a few example streaming configurations:
51
+
52
+- **Headless collector**:
53
+ - Child `A`, _without_ a database or web dashboard, streams metrics to parent `B`.
54
+ - `A` metrics are only available via the local Agent dashboard for `B`.
55
+ - `B` generates alarms for `A`.
56
+- **Replication**:
57
+ - Child `A`, _with_ a database and web dashboard, streams metrics to parent `B`.
58
+ - `A` metrics are available on both local Agent dashboards, and can be stored with the same or different metrics
59
+ retention policies.
60
+ - Both `A` and `B` generate alarms.
61
+- **Proxy**:
62
+ - Child `A`, _with or without_ a database, sends metrics to proxy `C`, also _with or without_ a database. `C` sends
63
+ metrics to parent `B`.
64
+ - Any node with a database can generate alarms.
65
+
66
+## Viewing streamed metrics
67
+
68
+Parent nodes feature a **Replicated Nodes** section in the left-hand panel, which opens with the hamburger icon
69
+
70
+in the top navigation. The parent node, plus any child nodes, appear here. Click on any of the hostnames to switch
71
+between parent and child dashboards, all served by the parent's [web server](/web/server/README.md).
72
+
73
+
75
+
76
+Each child dashboard is also available directly at the following URL pattern:
77
+`http://PARENT-NODE:19999/host/CHILD-HOSTNAME`.
78
+
79
+## What's next?
80
+
81
+Now that you understand the fundamentals of streaming metrics between nodes, go ahead and [enable
82
+streaming](/docs/metrics-storage-management/enable-streaming.mdx) using a simple `parent-child` relationship. For all
83
+the details, see the [streaming reference](/docs/metrics-storage-management/reference-streaming.mdx) doc.
84
+
85
+Take your streaming setup even further by [exporting metrics](/docs/export/external-databases.md) to an external TSDB.
86
+
87
+### Related information & further reading
88
+
89
+- Streaming
90
+ - **[How Netdata streams metrics](/docs/metrics-storage-management/how-streaming-works.mdx)**
91
+ - [Enable streaming between nodes](/docs/metrics-storage-management/enable-streaming.mdx)
92
+ - [Streaming reference](/docs/metrics-storage-management/reference-streaming.mdx)
\ No newline at end of file
docs/metrics-storage-management/reference-streaming.mdx
new
+486
@@ -0,0 +1,486 @@
1
+---
2
+title: "Streaming reference"
3
+description: "Each node running Netdata can stream the metrics it collects, in real time, to another node. See all of the available settings in this reference document."
4
+type: reference
5
+custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/metrics-storage-management/reference-streaming.mdx
6
+---
7
+
8
+# Streaming reference
9
+
10
+Each node running Netdata can stream the metrics it collects, in real time, to another node. To learn more, read about
11
+[how streaming works](/docs/metrics-storage-management/how-streaming-works.mdx).
12
+
13
+For a quickstart guide for enabling a simple `parent-child` streaming relationship, see our [stream metrics between
14
+nodes](/docs/metrics-storage-management/enable-streaming.mdx) doc. All other configuration options and scenarios are
15
+covered in the sections below.
16
+
17
+## Configuration
18
+
19
+There are two files responsible for configuring Netdata's streaming capabilities: `stream.conf` and `netdata.conf`.
20
+
21
+From within your Netdata config directory (typically `/etc/netdata`), [use `edit-config`](/docs/configure/nodes.md) to
22
+open either `stream.conf` or `netdata.conf`.
23
+
24
+```
25
+sudo ./edit-config stream.conf
26
+sudo ./edit-config netdata.conf
27
+```
28
+
29
+## Settings
30
+
31
+As mentioned above, both `stream.conf` and `netdata.conf` contain settings relevant to streaming.
32
+
33
+### `stream.conf`
34
+
35
+The `stream.conf` file contains three sections. The `[stream]` section is for configuring child nodes.
36
+
37
+The `[API_KEY]` and `[MACHINE_GUID]` sections are both for configuring parent nodes, and share the same settings.
38
+`[API_KEY]` settings affect every child node using that key, whereas `[MACHINE_GUID]` settings affect only the child
39
+node with a matching GUID.
40
+
41
+The file `/var/lib/netdata/registry/netdata.public.unique.id` contains a random GUID that **uniquely identifies each
42
+node**. This file is automatically generated by Netdata the first time it is started and remains unaltered forever.
43
+
44
+#### `[stream]` section
45
+
46
+| Setting | Default | Description |
47
+| :---------------------------------------------- | :------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
48
+| `enabled` | `no` | Whether this node streams metrics to any parent. Change to `yes` to enable streaming. |
49
+| [`destination`](#destination) | ` ` | A space-separated list of parent nodes to attempt to stream to, with the first available parent receiving metrics, using the following format: `[PROTOCOL:]HOST[%INTERFACE][:PORT][:SSL]`. [Read more →](#destination) |
50
+| `ssl skip certificate verification` | `yes` | If you want to accept self-signed or expired certificates, set to `yes` and uncomment. |
51
+| `CApath` | `/etc/ssl/certs/` | The directory where known certificates are found. Defaults to OpenSSL's default path. |
52
+| `CAfile` | `/etc/ssl/certs/cert.pem` | Add a parent node certificate to the list of known certificates in `CAPath`. |
53
+| `api key` | ` ` | The `API_KEY` to use as the child node. |
54
+| `timeout seconds` | `60` | The timeout to connect and send metrics to a parent. |
55
+| `default port` | `19999` | The port to use if `destination` does not specify one. |
56
+| [`send charts matching`](#send-charts-matching) | `*` | A space-separated list of [Netdata simple patterns](/libnetdata/simple_pattern/README.md) to filter which charts are streamed. [Read more →](#send-charts-matching) |
57
+| `buffer size bytes` | `1048576` | The size of the buffer to use when sending metrics. The default `1048576` equals a buffer of 1MB, which is good for 10-20 seconds of data. Increase this if you expect latencies higher than that. The buffer is flushed on reconnect. |
58
+| `reconnect delay seconds` | `5` | How long to wait until retrying to connect to the parent node. |
59
+| `initial clock resync iterations` | `60` | Sync the clock of charts for how many seconds when starting. |
60
+
61
+### `[API_KEY]` and `[MACHINE_GUID]` sections
62
+
63
+| Setting | Default | Description |
64
+| :---------------------------------------------- | :------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
65
+| `enabled` | `no` | Whether this API KEY enabled or disabled. |
66
+| [`allow from`](#allow-from) | `*` | A space-separated list of [Netdata simple patterns](/libnetdata/simple_pattern/README.md) matching the IPs of nodes that will stream metrics using this API key. [Read more →](#allow-from) |
67
+| `default history` | `3600` | The default amount of child metrics history to retain when using the `save`, `map`, or `ram` memory modes. |
68
+| [`default memory mode`](#default-memory-mode) | `ram` | The [database](/database/README.md) to use for all nodes using this `API_KEY`. Valid settings are `dbengine`, `map`, `save`, `ram`, or `none`. [Read more →](#default-memory-mode) |
69
+| `health enabled by default` | `auto` | Whether alarms and notifications should be enabled for nodes using this `API_KEY`. `auto` enables alarms when the child is connected. `yes` enables alarms always, and `no` disables alarms. |
70
+| `default postpone alarms on connect seconds` | `60` | Postpone alarms and notifications for a period of time after the child connects. |
71
+| `default proxy enabled` | ` ` | Route metrics through a proxy. |
72
+| `default proxy destination` | ` ` | Space-separated list of `IP:PORT` for proxies. |
73
+| `default proxy api key` | ` ` | The `API_KEY` of the proxy. |
74
+| `default send charts matching` | `*` | See [`send charts matching`](#send-charts-matching). |
75
+
76
+#### `destination`
77
+
78
+A space-separated list of parent nodes to attempt to stream to, with the first available parent receiving metrics, using
79
+the following format: `[PROTOCOL:]HOST[%INTERFACE][:PORT][:SSL]`.
80
+
81
+- `PROTOCOL`: `tcp`, `udp`, or `unix`. (only tcp and unix are supported by parent nodes)
82
+- `HOST`: A IPv4, IPv6 IP, or a hostname, or a unix domain socket path. IPv6 IPs should be given with brackets
83
+ `[ip:address]`.
84
+- `INTERFACE` (IPv6 only): The network interface to use.
85
+- `PORT`: The port number or service name (`/etc/services`) to use.
86
+- `SSL`: To enable TLS/SSL encryption of the streaming connection.
87
+
88
+To enable TCP streaming to a parent node at `203.0.113.0` on port `20000` and with TLS/SSL encryption:
89
+
90
+```conf
91
+[stream]
92
+ destination = tcp:203.0.113.0:20000:SSL
93
+```
94
+
95
+#### `send charts matching`
96
+
97
+A space-separated list of [Netdata simple patterns](/libnetdata/simple_pattern/README.md) to filter which charts are streamed.
98
+
99
+The default is a single wildcard `*`, which streams all charts.
100
+
101
+To send only a few charts, list them explicitly, or list a group using a wildcard. To send _only_ the `apps.cpu` chart
102
+and charts with contexts beginning with `system.`:
103
+
104
+```conf
105
+[stream]
106
+ send charts matching = apps.cpu system.*
107
+```
108
+
109
+To send all but a few charts, use `!` to create a negative match. To send _all_ charts _but_ `apps.cpu`:
110
+
111
+```conf
112
+[stream]
113
+ send charts matching = !apps.cpu *
114
+```
115
+
116
+#### `allow from`
117
+
118
+A space-separated list of [Netdata simple patterns](/libnetdata/simple_pattern/README.md) matching the IPs of nodes that
119
+will stream metrics using this API key. The order is important, left to right, as the first positive or negative match is used.
120
+
121
+The default is `*`, which accepts all requests including the `API_KEY`.
122
+
123
+To allow from only a specific IP address:
124
+
125
+```conf
126
+[API_KEY]
127
+ allow from = 203.0.113.10
128
+```
129
+
130
+To allow all IPs starting with `10.*`, except `10.1.2.3`:
131
+
132
+```conf
133
+[API_KEY]
134
+ allow from = !10.1.2.3 10.*
135
+```
136
+
137
+> If you set specific IP addresses here, and also use the `allow connections` setting in the `[web]` section of
138
+> `netdata.conf`, be sure to add the IP address there so that it can access the API port.
139
+
140
+#### `default memory mode`
141
+
142
+The [database](/database/README.md) to use for all nodes using this `API_KEY`. Valid settings are `dbengine`, `ram`,
143
+`save`, `map`, or `none`.
144
+
145
+- `dbengine`: The default, recommended time-series database (TSDB) for Netdata. Stores recent metrics in memory, then
146
+ efficiently spills them to disk for long-term storage.
147
+- `ram`: Stores metrics _only_ in memory, which means metrics are lost when Netdata stops or restarts. Ideal for
148
+ streaming configurations that use ephemeral nodes.
149
+- `save`: Stores metrics in memory, but saves metrics to disk when Netdata stops or restarts, and loads historical
150
+ metrics on start.
151
+- `map`: Stores metrics in memory-mapped files, like swap, with constant disk write.
152
+- `none`: No database.
153
+
154
+When using `default memory mode = dbengine`, the parent node creates a separate instance of the TSDB to store metrics
155
+from child nodes. The [size of _each_ instance is configurable](/docs/store/change-metrics-storage.md) with the `page
156
+cache size` and `dbengine multihost disk space` settings in the `[global]` section in `netdata.conf`.
157
+
158
+### `netdata.conf`
159
+
160
+| Setting | Default | Description |
161
+| :----------------------------------------- | :---------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
162
+| **`[global]` section** | | |
163
+| `memory mode` | `dbengine` | Determines the [database type](/database/README.md) to be used on that node. Other options settings include `none`, `ram`, `save`, and `map`. `none` disables the database at this host. This also disables alarms and notifications, as those can't run without a database. |
164
+| **`[web]` section** | | |
165
+| `mode` | `static-threaded` | Determines the [web server](/web/server/README.md) type. The other option is `none`, which disables the dashboard, API, and registry. |
166
+| `accept a streaming request every seconds` | `0` | Set a limit on how often a parent node accepts streaming requests from child nodes. `0` equals no limit. If this is set, you may see `... too busy to accept new streaming request. Will be allowed in X secs` in Netdata's `error.log`. |
167
+
168
+## Examples
169
+
170
+### Per-child settings
171
+
172
+While the `[API_KEY]` section applies settings for any child node using that key, you can also use per-child settings
173
+with the `[MACHINE_GUID]` section.
174
+
175
+For example, the metrics streamed from only the child node with `MACHINE_GUID` are saved in memory, not using the
176
+default `dbengine` as specified by the `API_KEY`, and alarms are disabled.
177
+
178
+```conf
179
+[API_KEY]
180
+ enabled = yes
181
+ default memory mode = dbengine
182
+ health enabled by default = auto
183
+ allow from = *
184
+
185
+[MACHINE_GUID]
186
+ enabled = yes
187
+ memory mode = save
188
+ health enabled = no
189
+```
190
+
191
+### Securing streaming with TLS/SSL
192
+
193
+Netdata does not activate TLS encryption by default. To encrypt streaming connections, you first need to [enable TLS
194
+support](/web/server/README.md#enabling-tls-support) on the parent. With encryption enabled on the receiving side, you
195
+need to instruct the child to use TLS/SSL as well. On the child's `stream.conf`, configure the destination as follows:
196
+
197
+```
198
+[stream]
199
+ destination = host:port:SSL
200
+```
201
+
202
+The word `SSL` appended to the end of the destination tells the child that connections must be encrypted.
203
+
204
+> While Netdata uses Transport Layer Security (TLS) 1.2 to encrypt communications rather than the obsolete SSL protocol,
205
+> it's still common practice to refer to encrypted web connections as `SSL`. Many vendors, like Nginx and even Netdata
206
+> itself, use `SSL` in configuration files, whereas documentation will always refer to encrypted communications as `TLS`
207
+> or `TLS/SSL`.
208
+
209
+#### Certificate verification
210
+
211
+When TLS/SSL is enabled on the child, the default behavior will be to not connect with the parent unless the server's
212
+certificate can be verified via the default chain. In case you want to avoid this check, add the following to the
213
+child's `stream.conf` file:
214
+
215
+```
216
+[stream]
217
+ ssl skip certificate verification = yes
218
+```
219
+
220
+#### Trusted certificate
221
+
222
+If you've enabled [certificate verification](#certificate-verification), you might see errors from the OpenSSL library
223
+when there's a problem with checking the certificate chain (`X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY`). More
224
+importantly, OpenSSL will reject self-signed certificates.
225
+
226
+Given these known issues, you have two options. If you trust your certificate, you can set the options `CApath` and
227
+`CAfile` to inform Netdata where your certificates, and the certificate trusted file, are stored.
228
+
229
+For more details about these options, you can read about [verify
230
+locations](https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_load_verify_locations.html).
231
+
232
+Before you changed your streaming configuration, you need to copy your trusted certificate to your child system and add
233
+the certificate to OpenSSL's list.
234
+
235
+On most Linux distributions, the `update-ca-certificates` command searches inside the `/usr/share/ca-certificates`
236
+directory for certificates. You should double-check by reading the `update-ca-certificate` manual (`man
237
+update-ca-certificate`), and then change the directory in the below commands if needed.
238
+
239
+If you have `sudo` configured on your child system, you can use that to run the following commands. If not, you'll have
240
+to log in as `root` to complete them.
241
+
242
+```
243
+# mkdir /usr/share/ca-certificates/netdata
244
+# cp parent_cert.pem /usr/share/ca-certificates/netdata/parent_cert.crt
245
+# chown -R netdata.netdata /usr/share/ca-certificates/netdata/
246
+```
247
+
248
+First, you create a new directory to store your certificates for Netdata. Next, you need to change the extension on your
249
+certificate from `.pem` to `.crt` so it's compatible with `update-ca-certificate`. Finally, you need to change
250
+permissions so the user that runs Netdata can access the directory where you copied in your certificate.
251
+
252
+Next, edit the file `/etc/ca-certificates.conf` and add the following line:
253
+
254
+```
255
+netdata/parent_cert.crt
256
+```
257
+
258
+Now you update the list of certificates running the following, again either as `sudo` or `root`:
259
+
260
+```
261
+# update-ca-certificates
262
+```
263
+
264
+> Some Linux distributions have different methods of updating the certificate list. For more details, please read this
265
+> guide on [adding trusted root certificates](https://github.com/Busindre/How-to-Add-trusted-root-certificates).
266
+
267
+Once you update your certificate list, you can set the stream parameters for Netdata to trust the parent certificate.
268
+Open `stream.conf` for editing and change the following lines:
269
+
270
+```
271
+[stream]
272
+ CApath = /etc/ssl/certs/
273
+ CAfile = /etc/ssl/certs/parent_cert.pem
274
+```
275
+
276
+With this configuration, the `CApath` option tells Netdata to search for trusted certificates inside `/etc/ssl/certs`.
277
+The `CAfile` option specifies the Netdata parent certificate is located at `/etc/ssl/certs/parent_cert.pem`. With this
278
+configuration, you can skip using the system's entire list of certificates and use Netdata's parent certificate instead.
279
+
280
+#### Expected behaviors
281
+
282
+With the introduction of TLS/SSL, the parent-child communication behaves as shown in the table below, depending on the
283
+following configurations:
284
+
285
+- **Parent TLS (Yes/No)**: Whether the `[web]` section in `netdata.conf` has `ssl key` and `ssl certificate`.
286
+- **Parent port TLS (-/force/optional)**: Depends on whether the `[web]` section `bind to` contains a `^SSL=force` or
287
+ `^SSL=optional` directive on the port(s) used for streaming.
288
+- **Child TLS (Yes/No)**: Whether the destination in the child's `stream.conf` has `:SSL` at the end.
289
+- **Child TLS Verification (yes/no)**: Value of the child's `stream.conf` `ssl skip certificate verification`
290
+ parameter (default is no).
291
+
292
+| Parent TLS enabled | Parent port SSL | Child TLS | Child SSL Ver. | Behavior |
293
+| :----------------- | :--------------- | :-------- | :------------- | :--------------------------------------------------------------------------------------------------------------------------------------- |
294
+| No | - | No | no | Legacy behavior. The parent-child stream is unencrypted. |
295
+| Yes | force | No | no | The parent rejects the child connection. |
296
+| Yes | -/optional | No | no | The parent-child stream is unencrypted (expected situation for legacy child nodes and newer parent nodes) |
297
+| Yes | -/force/optional | Yes | no | The parent-child stream is encrypted, provided that the parent has a valid TLS/SSL certificate. Otherwise, the child refuses to connect. |
298
+| Yes | -/force/optional | Yes | yes | The parent-child stream is encrypted. |
299
+
300
+### Proxy
301
+
302
+A proxy is a node that receives metrics from a child, then streams them onward to a parent. To configure a proxy,
303
+configure it as a receiving and a sending Netdata at the same time.
304
+
305
+Netdata proxies may or may not maintain a database for the metrics passing through them. When they maintain a database,
306
+they can also run health checks (alarms and notifications) for the remote host that is streaming the metrics.
307
+
308
+In the following example, the proxy receives metrics from a child node using the `API_KEY` of
309
+`66666666-7777-8888-9999-000000000000`, then stores metrics using `dbengine`. It then uses the `API_KEY` of
310
+`11111111-2222-3333-4444-555555555555` to proxy those same metrics on to a parent node at `203.0.113.0`.
311
+
312
+```conf
313
+[stream]
314
+ enabled = yes
315
+ destination = 203.0.113.0
316
+ api key = 11111111-2222-3333-4444-555555555555
317
+
318
+[66666666-7777-8888-9999-000000000000]
319
+ enabled = yes
320
+ default memory mode = dbengine
321
+```
322
+
323
+### Ephemeral nodes
324
+
325
+Netdata can help you monitor ephemeral nodes, such as containers in an auto-scaling infrastructure, by always streaming
326
+metrics to any number of permanently-running parent nodes.
327
+
328
+On the parent, set the following in `stream.conf`:
329
+
330
+```conf
331
+[11111111-2222-3333-4444-555555555555]
332
+ # enable/disable this API key
333
+ enabled = yes
334
+
335
+ # one hour of data for each of the child nodes
336
+ default history = 3600
337
+
338
+ # do not save child metrics on disk
339
+ default memory = ram
340
+
341
+ # alarms checks, only while the child is connected
342
+ health enabled by default = auto
343
+```
344
+
345
+On the child nodes, set the following in `stream.conf`:
346
+
347
+```bash
348
+[stream]
349
+ # stream metrics to another Netdata
350
+ enabled = yes
351
+
352
+ # the IP and PORT of the parent
353
+ destination = 10.11.12.13:19999
354
+
355
+ # the API key to use
356
+ api key = 11111111-2222-3333-4444-555555555555
357
+```
358
+
359
+In addition, edit `netdata.conf` on each child node to disable the database and alarms.
360
+
361
+```bash
362
+[global]
363
+ # disable the local database
364
+ memory mode = none
365
+
366
+[health]
367
+ # disable health checks
368
+ enabled = no
369
+```
370
+
371
+## Troubleshooting
372
+
373
+Both parent and child nodes log information at `/var/log/netdata/error.log`.
374
+
375
+If the child manages to connect to the parent you will see something like (on the parent):
376
+
377
+```
378
+2017-03-09 09:38:52: netdata: INFO : STREAM [receive from [10.11.12.86]:38564]: new client connection.
379
+2017-03-09 09:38:52: netdata: INFO : STREAM xxx [10.11.12.86]:38564: receive thread created (task id 27721)
380
+2017-03-09 09:38:52: netdata: INFO : STREAM xxx [receive from [10.11.12.86]:38564]: client willing to stream metrics for host 'xxx' with machine_guid '1234567-1976-11e6-ae19-7cdd9077342a': update every = 1, history = 3600, memory mode = ram, health auto
381
+2017-03-09 09:38:52: netdata: INFO : STREAM xxx [receive from [10.11.12.86]:38564]: initializing communication...
382
+2017-03-09 09:38:52: netdata: INFO : STREAM xxx [receive from [10.11.12.86]:38564]: receiving metrics...
383
+```
384
+
385
+and something like this on the child:
386
+
387
+```
388
+2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: connecting...
389
+2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: initializing communication...
390
+2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: waiting response from remote netdata...
391
+2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: established communication - sending metrics...
392
+```
393
+
394
+The following sections describe the most common issues you might encounter when connecting parent and child nodes.
395
+
396
+### Slow connections between parent and child
397
+
398
+When you have a slow connection between parent and child, Netdata raises a few different errors. Most of the
399
+errors will appear in the child's `error.log`.
400
+
401
+```bash
402
+netdata ERROR : STREAM_SENDER[CHILD HOSTNAME] : STREAM CHILD HOSTNAME [send to PARENT IP:PARENT PORT]: too many data pending - buffer is X bytes long,
403
+Y unsent - we have sent Z bytes in total, W on this connection. Closing connection to flush the data.
404
+```
405
+
406
+On the parent side, you may see various error messages, most commonly the following:
407
+
408
+```
409
+netdata ERROR : STREAM_PARENT[CHILD HOSTNAME,[CHILD IP]:CHILD PORT] : read failed: end of file
410
+```
411
+
412
+Another common problem in slow connections is the child sending a partial message to the parent. In this case, the
413
+parent will write the following to its `error.log`:
414
+
415
+```
416
+ERROR : STREAM_RECEIVER[CHILD HOSTNAME,[CHILD IP]:CHILD PORT] : sent command 'B' which is not known by netdata, for host 'HOSTNAME'. Disabling it.
417
+```
418
+
419
+In this example, `B` was part of a `BEGIN` message that was cut due to connection problems.
420
+
421
+Slow connections can also cause problems when the parent misses a message and then receives a command related to the
422
+missed message. For example, a parent might miss a message containing the child's charts, and then doesn't know
423
+what to do with the `SET` message that follows. When that happens, the parent will show a message like this:
424
+
425
+```
426
+ERROR : STREAM_RECEIVER[CHILD HOSTNAME,[CHILD IP]:CHILD PORT] : requested a SET on chart 'CHART NAME' of host 'HOSTNAME', without a dimension. Disabling it.
427
+```
428
+
429
+### Child cannot connect to parent
430
+
431
+When the child can't connect to a parent for any reason (misconfiguration, networking, firewalls, parent
432
+down), you will see the following in the child's `error.log`.
433
+
434
+```
435
+ERROR : STREAM_SENDER[HOSTNAME] : Failed to connect to 'PARENT IP', port 'PARENT PORT' (errno 113, No route to host)
436
+```
437
+
438
+### 'Is this a Netdata?'
439
+
440
+This question can appear when Netdata starts the stream and receives an unexpected response. This error can appear when
441
+the parent is using SSL and the child tries to connect using plain text. You will also see this message when
442
+Netdata connects to another server that isn't Netdata. The complete error message will look like this:
443
+
444
+```
445
+ERROR : STREAM_SENDER[CHILD HOSTNAME] : STREAM child HOSTNAME [send to PARENT HOSTNAME:PARENT PORT]: server is not replying properly (is it a netdata?).
446
+```
447
+
448
+### Stream charts wrong
449
+
450
+Chart data needs to be consistent between child and parent nodes. If there are differences between chart data on
451
+a parent and a child, such as gaps in metrics collection, it most often means your child's `memory mode`
452
+does not match the parent's. To learn more about the different ways Netdata can store metrics, and thus keep chart
453
+data consistent, read our [memory mode documentation](/database/README.md).
454
+
455
+### Forbidding access
456
+
457
+You may see errors about "forbidding access" for a number of reasons. It could be because of a slow connection between
458
+the parent and child nodes, but it could also be due to other failures. Look in your parent's `error.log` for errors
459
+that look like this:
460
+
461
+```
462
+STREAM [receive from [child HOSTNAME]:child IP]: `MESSAGE`. Forbidding access."
463
+```
464
+
465
+`MESSAGE` will have one of the following patterns:
466
+
467
+- `request without KEY` : The message received is incomplete and the KEY value can be API, hostname, machine GUID.
468
+- `API key 'VALUE' is not valid GUID`: The UUID received from child does not have the format defined in [RFC
469
+ 4122](https://tools.ietf.org/html/rfc4122)
470
+- `machine GUID 'VALUE' is not GUID.`: This error with machine GUID is like the previous one.
471
+- `API key 'VALUE' is not allowed`: This stream has a wrong API key.
472
+- `API key 'VALUE' is not permitted from this IP`: The IP is not allowed to use STREAM with this parent.
473
+- `machine GUID 'VALUE' is not allowed.`: The GUID that is trying to send stream is not allowed.
474
+- `Machine GUID 'VALUE' is not permitted from this IP. `: The IP does not match the pattern or IP allowed to connect to
475
+ use stream.
476
+
477
+### Netdata could not create a stream
478
+
479
+The connection between parent and child is a stream. When the parent can't convert the initial connection into
480
+a stream, it will write the following message inside `error.log`:
481
+
482
+```
483
+file descriptor given is not a valid stream
484
+```
485
+
486
+After logging this error, Netdata will close the stream.
docs/store/change-metrics-storage.md
+10
-7
@@ -8,9 +8,10 @@ custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/store/chang
8
9
import { Calculator } from '../../src/components/agent/dbCalc/'
10
11
-The [database engine](/database/engine/README.md) uses RAM to store recent metrics. When metrics reach a certain age,
12
-and based on how much system RAM you allocate toward storing metrics in memory, they are compressed and "spilled" to
13
-disk for long-term storage.
11
+The Netdata Agent uses a time-series database (TSDB), named the [database engine
12
+(`dbengine`)](/database/engine/README.md), to store metrics data. The most recently-collected metrics are stored in RAM,
13
+and when metrics reach a certain age, and based on how much system RAM you allocate toward storing metrics in memory,
14
+they are compressed and "spilled" to disk for long-term storage.
15
16
The default settings retain about two day's worth of metrics on a system collecting 2,000 metrics every second, but the
17
Netdata Agent is highly configurable if you want your nodes to store days, weeks, or months worth of per-second data.
@@ -59,15 +60,17 @@ method](/docs/configure/start-stop-restart.md) for your system, to change the da
60
61
## What's next?
62
62
-For more information about the database engine, see our [database reference doc](/database/engine/README.md).
63
+If you have multiple nodes with the Netdata Agent installed, you can [stream metrics](/docs/store/stream-metrics.md)
64
+from any number of _child_ nodes to a _parent_ node and store metrics using a centralized time-series database.Streaming
65
+allows you to centralize your data, run Agents as headless collectors, replicate data, and more.
66
67
Storing metrics with the database engine is completely interoperable with [exporting to other time-series
68
databases](/docs/export/external-databases.md). With exporting, you can use the node's resources to surface metrics
69
when [viewing dashboards](/docs/visualize/interact-dashboards-charts.md), while also archiving metrics elsewhere for
70
further analysis, visualization, or correlation with other tools.
71
69
-If you don't want to always store metrics on the node that collects them or run ephemeral nodes without dedicated
70
-storage, you can use [streaming](/streaming/README.md). Streaming allows you to centralize your data, run Agents as
71
-headless collectors, replicate data, and more.
72
+### Related reference documentation
73
+
74
+- [Netdata Agent · Database engine](/database/engine/README.md)
75
76
[](<>)