fixed README.md links (#4433)
Costa Tsaousis committed
Oct 18, 2018 at 19:01 UTC
37dc871a22a44763c5214fc56cae2bc8d59512fe
10 files changed
+126
-74
backends/README.md
+115
-53
@@ -1,76 +1,100 @@
1
2
-netdata supports backends for archiving the metrics, or providing long term dashboards, using grafana or other tools, like this:
2
+# Metrics Long Term Archiving
3
+
4
+netdata supports backends for archiving the metrics, or providing long term dashboards,
5
+using Grafana or other tools, like this:
6
7
8
6
-Since netdata collects thousands of metrics per server per second, which would easily congest any backend server when several netdata servers are sending data to it, netdata allows sending metrics at a lower frequency. So, although netdata collects metrics every second, it can send to the backend servers averages or sums every X seconds (though, it can send them per second if you need it to).
9
+Since netdata collects thousands of metrics per server per second, which would easily congest any backend
10
+server when several netdata servers are sending data to it, netdata allows sending metrics at a lower
11
+frequency, by resampling them.
12
+
13
+So, although netdata collects metrics every second, it can send to the backend servers averages or sums every
14
+X seconds (though, it can send them per second if you need it to).
15
16
## features
17
18
1. Supported backends
19
12
- 1. **graphite** (`plaintext interface`, used by **Graphite**, **InfluxDB**, **KairosDB**, **Blueflood**, **ElasticSearch** via logstash tcp input and the graphite codec, etc)
20
+ - **graphite** (`plaintext interface`, used by **Graphite**, **InfluxDB**, **KairosDB**,
21
+ **Blueflood**, **ElasticSearch** via logstash tcp input and the graphite codec, etc)
22
14
- metrics are sent to the backend server as `prefix.hostname.chart.dimension`. `prefix` is configured below, `hostname` is the hostname of the machine (can also be configured).
23
+ metrics are sent to the backend server as `prefix.hostname.chart.dimension`. `prefix` is
24
+ configured below, `hostname` is the hostname of the machine (can also be configured).
25
16
- 2. **opentsdb** (`telnet interface`, used by **OpenTSDB**, **InfluxDB**, **KairosDB**, etc)
26
+ - **opentsdb** (`telnet interface`, used by **OpenTSDB**, **InfluxDB**, **KairosDB**, etc)
27
18
- metrics are sent to opentsdb as `prefix.chart.dimension` with tag `host=hostname`.
28
+ metrics are sent to opentsdb as `prefix.chart.dimension` with tag `host=hostname`.
29
20
- 3. **json** document DBs
30
+ - **json** document DBs
31
22
- metrics are sent to a document db, `JSON` formatted.
32
+ metrics are sent to a document db, `JSON` formatted.
33
24
- 4. **prometheus** is described at [prometheus page](prometheus/) since it pulls data from netdata.
34
+ - **prometheus** is described at [prometheus page](prometheus/) since it pulls data from netdata.
35
36
2. Only one backend may be active at a time.
37
28
-3. All metrics are transferred to the backend - netdata does not implement any metric filtering.
38
+3. Netdata can filter metrics (at the chart level), to send only a subset of the collected metrics.
39
40
4. Three modes of operation (for all backends):
41
32
- 1. `as collected`: the latest collected value is sent to the backend. This means that if netdata is configured to send data to the backend every 10 seconds, only 1 out of 10 values will appear at the backend server. The values are sent exactly as collected, before any multipliers or dividers applied and before any interpolation. This mode emulates other data collectors, such as `collectd`.
42
+ - `as collected`: the latest collected value is sent to the backend. This means that if netdata
43
+ is configured to send data to the backend every 10 seconds, only 1 out of 10 values will appear
44
+ at the backend server. The values are sent exactly as collected, before any multipliers or
45
+ dividers applied and before any interpolation. This mode emulates other data collectors,
46
+ such as `collectd` or `telegraf`.
47
34
- 2. `average`: the average of the interpolated values shown on the netdata graphs is sent to the backend. So, if netdata is configured to send data to the backend server every 10 seconds, the average of the 10 values shown on the netdata charts will be used. **If you can't decide which mode to use, use `average`.**
48
+ - `average`: the average of the interpolated values shown on the netdata graphs is sent to the
49
+ backend. So, if netdata is configured to send data to the backend server every 10 seconds,
50
+ the average of the 10 values shown on the netdata charts will be used. **If you can't decide
51
+ which mode to use, use `average`.**
52
36
- 3. `sum` or `volume`: the sum of the interpolated values shown on the netdata graphs is sent to the backend. So, if netdata is configured to send data to the backend every 10 seconds, the sum of the 10 values shown on the netdata charts will be used.
53
+ - `sum` or `volume`: the sum of the interpolated values shown on the netdata graphs is sent to
54
+ the backend. So, if netdata is configured to send data to the backend every 10 seconds, the
55
+ sum of the 10 values shown on the netdata charts will be used.
56
57
5. This code is smart enough, not to slow down netdata, independently of the speed of the backend server.
58
59
## configuration
60
42
-In `/etc/netdata/netdata.conf` you should have something like this (if not download the latest version of `netdata.conf` from your netdata):
61
+In `/etc/netdata/netdata.conf` you should have something like this (if not download the latest version
62
+of `netdata.conf` from your netdata):
63
64
```
65
[backend]
46
- enabled = yes | no
47
- type = graphite | opentsdb | json
48
- host tags = list of TAG=VALUE
49
- destination = space separated list of [PROTOCOL:]HOST[:PORT] - the first working will be used
50
- data source = average | sum | as collected
51
- prefix = netdata
52
- hostname = my-name
53
- update every = 10
54
- buffer on failures = 10
55
- timeout ms = 20000
56
- send charts matching = *
57
- send hosts matching = localhost *
58
- send names instead of ids = yes
66
+ enabled = yes | no
67
+ type = graphite | opentsdb | json
68
+ host tags = list of TAG=VALUE
69
+ destination = space separated list of [PROTOCOL:]HOST[:PORT] - the first working will be used
70
+ data source = average | sum | as collected
71
+ prefix = netdata
72
+ hostname = my-name
73
+ update every = 10
74
+ buffer on failures = 10
75
+ timeout ms = 20000
76
+ send charts matching = *
77
+ send hosts matching = localhost *
78
+ send names instead of ids = yes
79
```
80
81
- `enabled = yes | no`, enables or disables sending data to a backend
82
83
- `type = graphite | opentsdb | json`, selects the backend type
84
65
-- `destination = host1 host2 host3 ...`, accepts **a space separated list** of hostnames, IPs (IPv4 and IPv6) and ports to connect to. Netdata will use the **first available** to send the metrics.
85
+- `destination = host1 host2 host3 ...`, accepts **a space separated list** of hostnames,
86
+ IPs (IPv4 and IPv6) and ports to connect to.
87
+ Netdata will use the **first available** to send the metrics.
88
89
The format of each item in this list, is: `[PROTOCOL:]IP[:PORT]`.
90
91
`PROTOCOL` can be `udp` or `tcp`. `tcp` is the default and only supported by the current backends.
92
71
- `IP` can be `XX.XX.XX.XX` (IPv4), or `[XX:XX...XX:XX]` (IPv6). For IPv6 you can to enclose the IP in `[]` to separate it from the port.
93
+ `IP` can be `XX.XX.XX.XX` (IPv4), or `[XX:XX...XX:XX]` (IPv6).
94
+ For IPv6 you can to enclose the IP in `[]` to separate it from the port.
95
73
- `PORT` can be a number of a service name. If omitted, the default port for the backend will be used (graphite = 2003, opentsdb = 4242).
96
+ `PORT` can be a number of a service name. If omitted, the default port for the backend will be used
97
+ (graphite = 2003, opentsdb = 4242).
98
99
Example IPv4:
100
@@ -84,45 +108,85 @@ In `/etc/netdata/netdata.conf` you should have something like this (if not downl
108
destination = [ffff:...:0001]:2003 10.11.12.1:2003
109
```
110
87
- When multiple servers are defined, netdata will try the next one when the first one fails. This allows you to load-balance different servers: give your backend servers in different order on each netdata.
111
+ When multiple servers are defined, netdata will try the next one when the first one fails. This allows
112
+ you to load-balance different servers: give your backend servers in different order on each netdata.
113
89
- netdata also ships [`nc-backend.sh`](https://github.com/netdata/netdata/blob/master/contrib/nc-backend.sh), a script that can be used as a fallback backend to save the metrics to disk and push them to the time-series database when it becomes available again. It can also be used to monitor / trace / debug the metrics netdata generates.
114
+ netdata also ships [`nc-backend.sh`](nc-backend.sh),
115
+ a script that can be used as a fallback backend to save the metrics to disk and push them to the
116
+ time-series database when it becomes available again. It can also be used to monitor / trace / debug
117
+ the metrics netdata generates.
118
91
-- `data source = as collected`, or `data source = average`, or `data source = sum`, selects the kind of data that will be sent to the backend.
119
+- `data source = as collected`, or `data source = average`, or `data source = sum`, selects the kind of
120
+ data that will be sent to the backend.
121
93
-- `hostname = my-name`, is the hostname to be used for sending data to the backend server. By default this is `[global].hostname`.
122
+- `hostname = my-name`, is the hostname to be used for sending data to the backend server. By default
123
+ this is `[global].hostname`.
124
125
- `prefix = netdata`, is the prefix to add to all metrics.
126
97
-- `update every = 10`, is the number of seconds between sending data to the backend. netdata will add some randomness to this number, to prevent stressing the backend server when many netdata servers send data to the same backend. This randomness does not affect the quality of the data, only the time they are sent.
98
-
99
-- `buffer on failures = 10`, is the number of iterations (each iteration is `[backend].update every` seconds) to buffer data, when the backend is not available. If the backend fails to receive the data after that many failures, data loss on the backend is expected (netdata will also log it).
100
-
101
-- `timeout ms = 20000`, is the timeout in milliseconds to wait for the backend server to process the data. By default this is `2 * update_every * 1000`.
102
-
103
-- `send hosts matching = localhost *` includes one or more space separated patterns, using ` * ` as wildcard (any number of times within each pattern). The patterns are checked against the hostname (the localhost is always checked as `localhost`), allowing us to filter which hosts will be sent to the backend when this netdata is a central netdata aggregating multiple hosts. A pattern starting with ` ! ` gives a negative match. So to match all hosts named `*db*` except hosts containing `*slave*`, use `!*slave* *db*` (so, the order is important: the first pattern matching the hostname will be used - positive or negative).
104
-
105
-- `send charts matching = *` includes one or more space separated patterns, using ` * ` as wildcard (any number of times within each pattern). The patterns are checked against both chart id and chart name. A pattern starting with ` ! ` gives a negative match. So to match all charts named `apps.*` except charts ending in `*reads`, use `!*reads apps.*` (so, the order is important: the first pattern matching the chart id or the chart name will be used - positive or negative).
106
-
107
-- `send names instead of ids = yes | no` controls the metric names netdata should send to backend. netdata supports names and IDs for charts and dimensions. Usually IDs are unique identifiers as read by the system and names are human friendly labels (also unique). Most charts and metrics have the same ID and name, but in several cases they are different: disks with device-mapper, interrupts, QoS classes, statsd synthetic charts, etc.
108
-
109
-- `host tags = list of TAG=VALUE` defines tags that should be appended on all metrics for the given host. These are currently only sent to opentsdb and prometheus. Please use the appropriate format for each time-series db. For example opentsdb likes them like `TAG1=VALUE1 TAG2=VALUE2`, but prometheus like `tag1="value1",tag2="value2"`. Host tags are mirrored with database replication (streaming of metrics between netdata servers).
127
+- `update every = 10`, is the number of seconds between sending data to the backend. netdata will add
128
+ some randomness to this number, to prevent stressing the backend server when many netdata servers send
129
+ data to the same backend. This randomness does not affect the quality of the data, only the time they
130
+ are sent.
131
+
132
+- `buffer on failures = 10`, is the number of iterations (each iteration is `[backend].update every` seconds)
133
+ to buffer data, when the backend is not available. If the backend fails to receive the data after that
134
+ many failures, data loss on the backend is expected (netdata will also log it).
135
+
136
+- `timeout ms = 20000`, is the timeout in milliseconds to wait for the backend server to process the data.
137
+ By default this is `2 * update_every * 1000`.
138
+
139
+- `send hosts matching = localhost *` includes one or more space separated patterns, using ` * ` as wildcard
140
+ (any number of times within each pattern). The patterns are checked against the hostname (the localhost
141
+ is always checked as `localhost`), allowing us to filter which hosts will be sent to the backend when
142
+ this netdata is a central netdata aggregating multiple hosts. A pattern starting with ` ! ` gives a
143
+ negative match. So to match all hosts named `*db*` except hosts containing `*slave*`, use
144
+ `!*slave* *db*` (so, the order is important: the first pattern matching the hostname will be used - positive
145
+ or negative).
146
+
147
+- `send charts matching = *` includes one or more space separated patterns, using ` * ` as wildcard (any
148
+ number of times within each pattern). The patterns are checked against both chart id and chart name.
149
+ A pattern starting with ` ! ` gives a negative match. So to match all charts named `apps.*`
150
+ except charts ending in `*reads`, use `!*reads apps.*` (so, the order is important: the first pattern
151
+ matching the chart id or the chart name will be used - positive or negative).
152
+
153
+- `send names instead of ids = yes | no` controls the metric names netdata should send to backend.
154
+ netdata supports names and IDs for charts and dimensions. Usually IDs are unique identifiers as read
155
+ by the system and names are human friendly labels (also unique). Most charts and metrics have the same
156
+ ID and name, but in several cases they are different: disks with device-mapper, interrupts, QoS classes,
157
+ statsd synthetic charts, etc.
158
+
159
+- `host tags = list of TAG=VALUE` defines tags that should be appended on all metrics for the given host.
160
+ These are currently only sent to opentsdb and prometheus. Please use the appropriate format for each
161
+ time-series db. For example opentsdb likes them like `TAG1=VALUE1 TAG2=VALUE2`, but prometheus like
162
+ `tag1="value1",tag2="value2"`. Host tags are mirrored with database replication (streaming of metrics
163
+ between netdata servers).
164
165
## monitoring operation
166
167
netdata provides 5 charts:
168
115
-1. **Buffered metrics**, the number of metrics netdata added to the buffer for dispatching them to the backend server.
169
+1. **Buffered metrics**, the number of metrics netdata added to the buffer for dispatching them to the
170
+ backend server.
171
+
172
2. **Buffered data size**, the amount of data (in KB) netdata added the buffer.
117
-3. ~~**Backend latency**, the time the backend server needed to process the data netdata sent. If there was a re-connection involved, this includes the connection time.~~ (this chart has been removed, because it only measures the time netdata needs to give the data to the O/S - since the backend servers do not ack the reception, netdata does not have any means to measure this properly)
173
+
174
+3. ~~**Backend latency**, the time the backend server needed to process the data netdata sent.
175
+ If there was a re-connection involved, this includes the connection time.~~
176
+ (this chart has been removed, because it only measures the time netdata needs to give the data
177
+ to the O/S - since the backend servers do not ack the reception, netdata does not have any means
178
+ to measure this properly).
179
+
180
4. **Backend operations**, the number of operations performed by netdata.
119
-5. **Backend thread CPU usage**, the CPU resources consumed by the netdata thread, that is responsible for sending the metrics to the backend server.
181
+
182
+5. **Backend thread CPU usage**, the CPU resources consumed by the netdata thread, that is responsible
183
+ for sending the metrics to the backend server.
184
185
186
187
## alarms
188
125
-The latest version of the alarms configuration for monitoring the backend is here: https://github.com/netdata/netdata/blob/master/conf.d/health.d/backend.conf
189
+The latest version of the alarms configuration for monitoring the backend is [here](../health/health.d/backend.conf)
190
191
netdata adds 4 alarms:
192
@@ -133,5 +197,3 @@ netdata adds 4 alarms:
197
198
199
136
-## InfluxDB setup as netdata backend (example)
137
-You can find blog post with example: how to use InfluxDB with netdata [here](https://blog.hda.me/2017/01/09/using-netdata-with-influxdb-backend.html)
collectors/charts.d.plugin/ap/README.md
-2
@@ -2,8 +2,6 @@
2
3
The `ap` collector visualizes data related to access points.
4
5
-The source code is [here](https://github.com/netdata/netdata/blob/master/charts.d/ap.chart.sh).
6
-
5
## Example netdata charts
6
7
collectors/node.d.plugin/named/README.md
-2
@@ -2,8 +2,6 @@
2
3
Using this netdata collector, you can monitor one or more ISC Bind servers.
4
5
-The source code for this plugin in [here](https://github.com/netdata/netdata/blob/master/node.d/named.node.js).
6
-
5
## Example netdata charts
6
7
Depending on the number of views your bind has, you may get a large number of charts.
collectors/node.d.plugin/sma_webbox/README.md
+1
-1
@@ -1,5 +1,5 @@
1
2
-[SMA Sunny Webbox](http://www.solar-is-future.com/sma-technology-for-our-future/products/sunny-webbox/index.html)
2
+[SMA Sunny Webbox](http://files.sma.de/dl/4253/WEBBOX-DUS131916W.pdf)
3
4
Example netdata configuration for node.d/sma_webbox.conf
5
collectors/node.d.plugin/snmp/README.md
+3
-5
@@ -10,8 +10,6 @@ This collector supports:
10
- each SNMP device may have a different update frequency
11
- each SNMP device will accept one or more batches to report values (you can set `max_request_size` per SNMP server, to control the size of batches).
12
13
-The source code of the plugin is [here](https://github.com/netdata/netdata/blob/master/node.d/snmp.node.js).
14
-
13
## Configuration
14
15
You will need to create the file `/etc/netdata/node.d/snmp.conf` with data like the following.
@@ -23,7 +21,7 @@ In this example:
21
- we will update the values every 10 seconds (`update_every: 10` under the server `10.11.12.8`).
22
- we define 2 charts `snmp_switch.bandwidth_port1` and `snmp_switch.bandwidth_port2`, each having 2 dimensions: `in` and `out`.
23
26
-```js
24
+```json
25
{
26
"enable_autodetect": false,
27
"update_every": 5,
@@ -105,7 +103,7 @@ Each of the 24 new charts will have its id (1-24) appended at:
103
3. its `oid` (for all dimensions), i.e. dimension `in` will be `1.3.6.1.2.1.2.2.1.10.1` to `1.3.6.1.2.1.2.2.1.10.24`
104
3. its priority (which will be incremented for each chart so that the charts will appear on the dashboard in this order)
105
108
-```js
106
+```json
107
{
108
"enable_autodetect": false,
109
"update_every": 10,
@@ -210,7 +208,7 @@ This switch also reports various other metrics, like snmp, packets per port, etc
208
209
This switch has a very slow SNMP processors. To respond, it needs about 8 seconds, so I have set the refresh frequency (`update_every`) to 15 seconds.
210
213
-```js
211
+```json
212
{
213
"enable_autodetect": false,
214
"update_every": 5,
collectors/node.d.plugin/stiebeleltron/README.md
+1
-3
@@ -46,8 +46,6 @@ The charts are configurable, however, the provided default configuration collect
46
47
### configuration
48
49
-The default configuration is provided in [netdata/conf.d/node.d/stiebeleltron.conf.md](https://github.com/netdata/netdata/blob/master/conf.d/node.d/stiebeleltron.conf.md). Just change the `update_every` (if necessary) and hostnames. **You may have to adapt the configuration to suit your needs and setup** (which might be different).
50
-
49
If no configuration is given, the module will be disabled. Each `update_every` is optional, the default is `10`.
50
51
---
@@ -78,7 +76,7 @@ In my case, the ISG is relatively slow with responding (at least 1s, but also up
76
* The dimensions support variable digits, the default is `1`. Most of the values printed by ISG are using 1 digit, some use 2.
77
* The dimensions also support the `multiplier` and `divisor` attributes, however the divisor gets overridden by `digits`, if specified. Default is `1`.
78
* The test string for the regex is always the whole HTML output from the url. For each parameter you need to have a regular expression that extracts the value from the HTML source in the first capture group.
81
- Recommended: [regexr.com](regexr.com) for testing and matching, [freeformatter.com](https://www.freeformatter.com/json-escape.html) for escaping the newly created regex for the JSON config.
79
+ Recommended: [regexr.com](https://regexr.com/) for testing and matching, [freeformatter.com](https://www.freeformatter.com/json-escape.html) for escaping the newly created regex for the JSON config.
80
81
The charts are being generated using the configuration below. So if your installation is in another language or has other metrics, just adapt the structure or regexes.
82
### Configuration template
collectors/plugins.d/README.md
+3
-5
@@ -374,15 +374,15 @@ or do not output the line at all.
374
375
## Modular Plugins
376
377
-1. **python**, use `python.d.plugin`, there are many examples in the [python.d directory](https://github.com/netdata/netdata/tree/master/python.d)
377
+1. **python**, use `python.d.plugin`, there are many examples in the [python.d directory](../python.d.plugin)
378
379
python is ideal for netdata plugins. It is a simple, yet powerful way to collect data, it has a very small memory footprint, although it is not the most CPU efficient way to do it.
380
381
-2. **node.js**, use `node.d.plugin`, there are a few examples in the [node.d directory](https://github.com/netdata/netdata/tree/master/node.d)
381
+2. **node.js**, use `node.d.plugin`, there are a few examples in the [node.d directory](../node.d.plugin)
382
383
node.js is the fastest scripting language for collecting data. If your plugin needs to do a lot of work, compute values, etc, node.js is probably the best choice before moving to compiled code. Keep in mind though that node.js is not memory efficient; it will probably need more RAM compared to python.
384
385
-3. **BASH**, use `charts.d.plugin`, there are many examples in the [charts.d directory](https://github.com/netdata/netdata/tree/master/charts.d)
385
+3. **BASH**, use `charts.d.plugin`, there are many examples in the [charts.d directory](../charts.d.plugin)
386
387
BASH is the simplest scripting language for collecting values. It is the less efficient though in terms of CPU resources. You can use it to collect data quickly, but extensive use of it might use a lot of system resources.
388
@@ -390,8 +390,6 @@ or do not output the line at all.
390
391
Of course, C is the most efficient way of collecting data. This is why netdata itself is written in C.
392
393
-5. **Nim**, there is an unofficial [nim plugin helper](https://github.com/FedericoCeratto/nim-netdata-plugin)
394
-
393
---
394
395
## Writing Plugins Properly
collectors/python.d.plugin/sensors/README.md
+1
-1
@@ -6,7 +6,7 @@ Charts are created dynamically.
6
7
### configuration
8
9
-For detailed configuration information please read [`sensors.conf`](https://github.com/netdata/netdata/blob/master/conf.d/python.d/sensors.conf) file.
9
+For detailed configuration information please read [`sensors.conf`](sensors.conf) file.
10
11
### possible issues
12
collectors/python.d.plugin/springboot/README.md
+1
-1
@@ -126,4 +126,4 @@ You can disable the default charts by set `defaults.<chart-id>: false`.
126
127
The dimension name of extras charts should replace `.` to `_`.
128
129
-Please check [springboot.conf](https://github.com/netdata/netdata/blob/master/conf.d/python.d/springboot.conf) for more examples.
\ No newline at end of file
129
+Please check [springboot.conf](springboot.conf) for more examples.
\ No newline at end of file
collectors/python.d.plugin/w1sensor/README.md
+1
-1
@@ -8,6 +8,6 @@ Charts are created dynamically based on the number of detected sensors.
8
9
### configuration
10
11
-For detailed configuration information please read [`w1sensor.conf`](https://github.com/netdata/netdata/blob/master/conf.d/python.d/w1sensor.conf) file.
11
+For detailed configuration information please read [`w1sensor.conf`](w1sensor.conf) file.
12
13
---