Tutorials to support v1.20 release (#7943)
* Add draft of CockroachDB tutorial * Fixed and new images * Support figures for images * Change border color * Change job * Initialize eBPF tutorial * Very very rough draft of host labels tutorial * Add a few mentions of tutorial * Fix for Thiago * Simplify health entities * Fixes for Thiago * Fixes and add tutorials to collectors README * Fixes to cockroachBD * Remove ebpf tutorial * remove link * Updates for Patti and Thiago * Add streaming security note * Straightaway
Joel Hans committed
Feb 20, 2020 at 20:46 UTC
19ef3f93792d382992afbdce74743d868eaa3074
6 files changed
+360
-5
collectors/README.md
+2
@@ -26,6 +26,8 @@ guide](QUICKSTART.md).
26
27
[Monitor Nginx or Apache web server log files with Netdata](../docs/tutorials/collect-apache-nginx-web-logs.md)
28
29
+[Monitor CockroadchDB metrics with Netdata](../docs/tutorials/monitor-cockroachdb.md)
30
+
31
[Monitor Unbound DNS servers with Netdata](../docs/tutorials/collect-unbound-metrics.md)
32
33
[Monitor a Hadoop cluster with Netdata](../docs/tutorials/monitor-hadoop-cluster.md)
docs/configuration-guide.md
+6
-1
@@ -187,7 +187,12 @@ So, Netdata supports [simple patterns](../libnetdata/simple_pattern/).
187
188
## Netdata labels
189
190
-Since version 1.20, Netdata accepts user defined labels for host. The labels are defined in the section `[host labels]`.
190
+Beginning with 1.20, Netdata accepts user-defined **host labels**. These labels are defined in the section `[host
191
+labels]`.
192
+
193
+Read more about how these labels work and why they're an effective way to organize complex infrasturctures in our
194
+tutorial: [Use host labels to organize systems, metrics, and alarms](tutorials/using-host-labels.md).
195
+
196
To define a label inside this section, some rules needs to be followed, or Netdata will reject the label. The following
197
restrictions are applied for label names:
198
docs/generator/custom/css/netdata.css
+16
-1
@@ -429,6 +429,21 @@ html [data-md-color-primary="blue-grey"] .md-nav--primary .md-nav__title--site {
429
font-size: 0.75rem;
430
}
431
432
+/* Support figures and figcaptions with some styling. */
433
+.md-typeset figure img {
434
+ border-radius: 3px 3px 0 0;
435
+}
436
+
437
+.md-typeset figure figcaption {
438
+ font-size: 0.75rem;
439
+ font-style: italic;
440
+ margin-top: -8px;
441
+ padding: 0.5rem;
442
+ border-width: 0 1px 1px 1px;
443
+ border-color: #AEB3B7;
444
+ border-style: solid;
445
+}
446
+
447
/*
448
Installer grid
449
*/
@@ -443,7 +458,7 @@ html [data-md-color-primary="blue-grey"] .md-nav--primary .md-nav__title--site {
458
.grid-item {
459
grid-column: span 2;
460
border-radius: 2px;
446
- border: 1px solid black;
461
+ border: 1px solid #AEB3B7;
462
padding: 1rem;
463
}
464
docs/tutorials/monitor-cockroachdb.md
new
+128
@@ -0,0 +1,128 @@
1
+# Monitor CockroachDB metrics with Netdata
2
+
3
+[CockroachDB](https://github.com/cockroachdb/cockroach) is an open-source project that brings SQL databases into
4
+scalable, disaster-resilient cloud deployments. Thanks to a [new CockroachDB
5
+collector](https://docs.netdata.cloud/collectors/go.d.plugin/modules/cockroachdb/) released in
6
+[v1.20](https://blog.netdata.cloud/posts/release-1.20/), you can now monitor any number of CockroachDB databases with
7
+maximum granularity using Netdata. Collect more than 50 unique metrics and put them on interactive visualizations
8
+designed for better visual anomaly detection.
9
+
10
+Netdata itself uses CockroachDB as part of its Netdata Cloud infrastructure, so we're happy to introduce this new
11
+collector and help others get started with it straightaway.
12
+
13
+Let's dive in and walk through the process of monitoring CockroachDB metrics with Netdata.
14
+
15
+## What's in this guide
16
+
17
+- [Configure the CockroachDB collector](#configure-the-cockroachdb-collector)
18
+ - [Manual setup for a local CockroachDB database](#manual-setup-for-a-local-cockroachdb-database)
19
+- [Tweak CockroachDB alarms](#tweak-cockroachdb-alarms)
20
+
21
+## Configure the CockroachDB collector
22
+
23
+Because _all_ of Netdata's collectors can auto-detect the services they monitor, you _shouldn't_ need to worry about
24
+configuring CockroachDB. Netdata only needs to regularly query the database's `_status/vars` page to gather metrics and
25
+display them on the dashboard.
26
+
27
+If your CockroachDB instance is accessible through `http://localhost:8080/` or `http://127.0.0.1:8080`, your setup is
28
+complete. Restart Netdata with `service netdata restart`, or use the [appropriate
29
+method](../getting-started.md#start-stop-and-restart-netdata) for your system, and refresh your browser. You should see
30
+CockroachDB metrics in your Netdata dashboard!
31
+
32
+<figure>
33
+ <img src="https://user-images.githubusercontent.com/1153921/73564467-d7e36b00-441c-11ea-9ec9-b5d5ea7277d4.png" alt="CPU utilization charts from a CockroachDB database monitored by Netdata">
34
+ <figcaption>CPU utilization charts from a CockroachDB database monitored by Netdata</figcaption>
35
+</figure>
36
+
37
+> Note: Netdata collects metrics from CockroachDB every 10 seconds, instead of our usual 1 second, because CockroachDB
38
+> only updates `_status/vars` every 10 seconds. You can't change this setting in CockroachDB.
39
+
40
+If you don't see CockroachDB charts, you may need to configure the collector manually.
41
+
42
+### Manual setup for a local CockroachDB database
43
+
44
+To configure Netdata's CockroachDB collector, navigate to your Netdata configuration directory (typically at
45
+`/etc/netdata/`) and use `edit-config` to initialize and edit your CockroachDB configuration file.
46
+
47
+```bash
48
+cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/
49
+./edit-config go.d/cockroachdb.conf
50
+```
51
+
52
+Scroll down to the `[JOBS]` section at the bottom of the file. You will see the two default jobs there, which you can
53
+edit, or create a new job with any of the parameters listed above in the file. Both the `name` and `url` values are
54
+required, and everything else is optional.
55
+
56
+For a production cluster, you'll use either an IP address or the system's hostname. Be sure that your remote system
57
+allows TCP communication on port 8080, or whichever port you have configured CockroachDB's [Admin
58
+UI](https://www.cockroachlabs.com/docs/stable/monitoring-and-alerting.html#prometheus-endpoint) to listen on.
59
+
60
+```yaml
61
+# [ JOBS ]
62
+jobs:
63
+ - name: remote
64
+ url: http://203.0.113.0:8080/_status/vars
65
+
66
+ - name: remote_hostname
67
+ url: http://cockroachdb.example.com:8080/_status/vars
68
+```
69
+
70
+For a secure cluster, use `https` in the `url` field instead.
71
+
72
+```yaml
73
+# [ JOBS ]
74
+jobs:
75
+ - name: remote
76
+ url: https://203.0.113.0:8080/_status/vars
77
+ tls_skip_verify: yes # If your certificate is self-signed
78
+
79
+ - name: remote_hostname
80
+ url: https://cockroachdb.example.com:8080/_status/vars
81
+ tls_skip_verify: yes # If your certificate is self-signed
82
+```
83
+
84
+You can add as many jobs as you'd like based on how many CockroachDB databases you have—Netdata will create separate
85
+charts for each job. Once you've edited `cockroachdb.conf` according to the needs of your infrastructure, restart
86
+Netdata to see your new charts.
87
+
88
+<figure>
89
+ <img src="https://user-images.githubusercontent.com/1153921/73564469-d7e36b00-441c-11ea-8333-02ba0e1c294c.png" alt="Charts showing a node failure during a simulated test">
90
+ <figcaption>Charts showing a node failure during a simulated test</figcaption>
91
+</figure>
92
+
93
+## Tweak CockroachDB alarms
94
+
95
+This release also includes eight pre-configured alarms for live nodes, such as whether the node is live, storage
96
+capacity, issues with replication, and the number of SQL connections/statements. See [health.d/cockroachdb.conf on
97
+GitHub](https://raw.githubusercontent.com/netdata/netdata/master/health/health.d/cockroachdb.conf) for details.
98
+
99
+You can also edit these files directly with `edit-config`:
100
+
101
+```bash
102
+cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/
103
+./edit-config health.d/cockroachdb.conf # You may need to use `sudo` for write privileges
104
+```
105
+
106
+For more information about editing the defaults or writing new alarm entities, see our health monitoring [quickstart
107
+guide](../../health/QUICKSTART.md).
108
+
109
+## What's next?
110
+
111
+Now that you're collecting metrics from your CockroachDB databases, let us know how it's working for you! There's always
112
+room for improvement or refinement based on real-world use cases. Feel free to [file an
113
+issue](https://github.com/netdata/netdata/issues/new?labels=bug%2C+needs+triage&template=bug_report.md) with your
114
+thoughts.
115
+
116
+Also, be sure to check out these useful resources:
117
+
118
+- [Netdata's CockroachDB documentation](https://docs.netdata.cloud/collectors/go.d.plugin/modules/cockroachdb/)
119
+- [Netdata's CockroachDB
120
+ configuration](https://github.com/netdata/go.d.plugin/blob/master/config/go.d/cockroachdb.conf)
121
+- [Netdata's CockroachDB
122
+ alarms](https://github.com/netdata/netdata/blob/29d9b5e51603792ee27ef5a21f1de0ba8e130158/health/health.d/cockroachdb.conf)
123
+- [CockroachDB homepage](https://www.cockroachlabs.com/product/)
124
+- [CockroachDB documentation](https://www.cockroachlabs.com/docs/stable/)
125
+- [`_status/vars` endpoint
126
+ docs](https://www.cockroachlabs.com/docs/stable/monitoring-and-alerting.html#prometheus-endpoint)
127
+- [Monitor CockroachDB with
128
+ Prometheus](https://www.cockroachlabs.com/docs/stable/monitor-cockroachdb-with-prometheus.html)
docs/tutorials/using-host-labels.md
new
+203
@@ -0,0 +1,203 @@
1
+# Use host labels to organize systems, metrics, and alarms
2
+
3
+When you use Netdata to monitor and troubleshoot an entire infrastructure, whether that's dozens or hundreds of systems,
4
+you need sophisticated ways of keeping everything organized. You need alarms that adapt to the system's purpose, or
5
+whether the `master` or `slave` in a streaming setup. You need properly-labeled metrics archiving so you can sort,
6
+correlate, and mash-up your data to your heart's content. You need to keep tabs on ephemeral Docker containers in a
7
+Kubernetes cluster.
8
+
9
+You need **host labels**: a powerful new way of organizing your Netdata-monitored systems. We introduced host labels in
10
+[v1.20 of Netdata](https://blog.netdata.cloud/posts/release-1.20/), and they come pre-configured out of the box.
11
+
12
+Let's take a peek into how to create host labels and apply them across a few of Netdata's features to give you more
13
+organization power over your infrastructure.
14
+
15
+## Create unique host labels
16
+
17
+Host labels are defined in `netdata.conf`. To create host labels, open that file using `edit-config`.
18
+
19
+```bash
20
+cd /etc/netdata # Replace this path with your Netdata config directory, if different
21
+sudo ./edit-config netdata.conf
22
+```
23
+
24
+Create a new `[host labels]` section defining a new host label and its value for the system in question. Make sure not
25
+to violate any of the [host label naming rules](../configuration-guide.md#netdata-labels).
26
+
27
+```conf
28
+[host labels]
29
+ type = webserver
30
+ location = us-seattle
31
+ installed = 20200218
32
+```
33
+
34
+Once you've written a few host labels, you need to enable them. Instead of restarting the entire Netdata service, you
35
+can reload labels using the helpful `netdatacli` tool:
36
+
37
+```bash
38
+netdatacli reload-labels
39
+```
40
+
41
+Your host labels will now be enabled. You can double-check these by using `curl http://HOST-IP:19999/api/v1/info` to
42
+read the status of your agent. For example, from a VPS system running Debian 10:
43
+
44
+```json
45
+{
46
+ ...
47
+ "host_labels": {
48
+ "_is_master": "false",
49
+ "_virt_detection": "systemd-detect-virt",
50
+ "_container_detection": "none",
51
+ "_container": "unknown",
52
+ "_virtualization": "kvm",
53
+ "_architecture": "x86_64",
54
+ "_kernel_version": "4.19.0-6-amd64",
55
+ "_os_version": "10 (buster)",
56
+ "_os_name": "Debian GNU/Linux",
57
+ "type": "webserver",
58
+ "location": "seattle",
59
+ "installed": "20200218"
60
+ },
61
+ ...
62
+}
63
+```
64
+
65
+You may have noticed a handful of labels that begin with an underscore (`_`). These are automatic labels.
66
+
67
+### Automatic labels
68
+
69
+When Netdata starts, it captures relevant information about the system and converts them into automatically-generated
70
+host labels. You can use these to logically organize your systems via health entities, exporting metrics,
71
+streaming/master status, and more.
72
+
73
+They capture the following:
74
+
75
+- Kernel version
76
+- Operating system name and version
77
+- CPU architecture, system cores, CPU frequency, RAM, and disk space
78
+- Whether Netdata is running inside of a container, and if so, the OS and hardware details about the container's host
79
+- What virtualization layer the system runs on top of, if any
80
+- Whether the system is a streaming master or slave
81
+
82
+If you want to organize your systems without manually creating host tags, try the automatic labels in some of the
83
+features below.
84
+
85
+## Host labels in streaming
86
+
87
+You may have noticed the `_is_master` and `_is_slave` automatic labels from above. Host labels are also now streamed
88
+from a slave to its master agent, which concentrates an entire infrastructure's OS, hardware, container, and
89
+virtualization information in one place: the master.
90
+
91
+Now, if you'd like to remind yourself of how much RAM a certain slave system has, you can simply access
92
+`http://localhost:19999/host/SLAVE_NAME/api/v1/info` and reference the automatically-generated host labels from the
93
+slave system. It's a vastly simplified way of accessing critical information about your infrastructure.
94
+
95
+> ⚠️ Because automatic labels for slave nodes are accessible via API calls, and contain sensitive information like
96
+> kernel and operating system versions, you should secure streaming connections with SSL. See the [streaming
97
+> documentation](../..//streaming/README.md#securing-streaming-communications) for details. You may also want to use
98
+> [access lists](../../web/server/README.md#access-lists) or [expose the API only to LAN/localhost
99
+> connections](../netdata-security.md#expose-netdata-only-in-a-private-lan).
100
+
101
+You can also use `_is_master`, `_is_slave`, and any other host labels in both health entities and metrics exporting.
102
+Speaking of which...
103
+
104
+## Host labels in health entities
105
+
106
+You can use host labels to logically organize your systems by their type, purpose, or location, and then apply specific
107
+alarms to them.
108
+
109
+For example, let's use configuration example from earlier:
110
+
111
+```conf
112
+[host labels]
113
+ type = webserver
114
+ location = us-seattle
115
+ installed = 20200218
116
+```
117
+
118
+You could now create a new health entity (checking if disk space will run out soon) that applies only to any host
119
+labeled `webserver`:
120
+
121
+```yaml
122
+ template: disk_fill_rate
123
+ on: disk.space
124
+ lookup: max -1s at -30m unaligned of avail
125
+ calc: ($this - $avail) / (30 * 60)
126
+ every: 15s
127
+ host labels: type = webserver
128
+```
129
+
130
+Or, by using one of the automatic labels, for only webserver systems running a specific OS:
131
+
132
+```yaml
133
+ host labels: _os_name = Debian*
134
+```
135
+
136
+In a streaming configuration where a master agent is triggering alarms for its slaves, you could create health entities
137
+that apply only to slaves:
138
+
139
+```yaml
140
+ host labels: _is_slave = true
141
+```
142
+
143
+Or when ephemeral Docker nodes are involved:
144
+
145
+```yaml
146
+ host labels: _container = docker
147
+```
148
+
149
+Of course, there are many more possibilities for intuitively organizing your systems with host labels. See the [health
150
+documentation](../../health/REFERENCE.md#alarm-line-host-labels) for more details, and then get creative!
151
+
152
+## Host labels in metrics exporting
153
+
154
+If you have enabled any metrics exporting via our experimental [exporters](../../exporting/README.md), any new host
155
+labels you created manually are sent to the destination database alongside metrics. You can change this behavior by
156
+editing `exporting.conf`, and you can even send automatically-generated labels on with exported metrics.
157
+
158
+```conf
159
+[exporting:global]
160
+enabled = yes
161
+send configured labels = yes
162
+send automatic labels = no
163
+```
164
+
165
+You can also change this behavior per exporting connection:
166
+
167
+```conf
168
+[opentsdb:my_instance3]
169
+enabled = yes
170
+destination = localhost:4242
171
+data source = sum
172
+update every = 10
173
+send charts matching = system.cpu
174
+send configured labels = no
175
+send automatic labels = yes
176
+```
177
+
178
+By applying labels to exported metrics, you can more easily parse historical metrics with the labels applied. To learn
179
+more about exporting, read the [documentation](../../exporting/README.md).
180
+
181
+## What's next?
182
+
183
+Host labels are a brand-new feature to Netdata, and yet they've already propagated deeply into some of its core
184
+functionality. We're just getting started with labels, and will keep the community apprised of additional functionality
185
+as it's made available. You can also track [issue #6503](https://github.com/netdata/netdata/issues/6503), which is where
186
+the Netdata team first kicked off this work.
187
+
188
+It should be noted that while the Netdata dashboard does not expose either user-configured or automatic host labels, API
189
+queries _do_ showcase this information. As always, we recommend you secure Netdata
190
+
191
+- [Expose Netdata only in a private LAN](../netdata-security.md#expose-netdata-only-in-a-private-lan)
192
+- [Enable TLS/SSL for web/API requests](../../web/server/README.md#enabling-tls-support)
193
+- Put Netdata behind a proxy
194
+ - [Use an authenticating web server in proxy
195
+ mode](../netdata-security.md#use-an-authenticating-web-server-in-proxy-mode)
196
+ - [Nginx proxy](../Running-behind-nginx.md)
197
+ - [Apache proxy](../Running-behind-apache.md)
198
+ - [Lighttpd](../Running-behind-lighttpd.md)
199
+ - [Caddy](../Running-behind-caddy.md)
200
+
201
+If you have issues or questions around using host labels, don't hesitate to [file an
202
+issue](https://github.com/netdata/netdata/issues/new?labels=bug%2C+needs+triage&template=bug_report.md) on GitHub. We're
203
+excited to make host labels even more valuable to our users, which we can only do with your input.
health/REFERENCE.md
+5
-3
@@ -374,8 +374,10 @@ good idea to tell Netdata to not clear the notification, by using the `no-clear-
374
375
#### Alarm line `host labels`
376
377
-Defines the list of labels present on a host. For example, let's suppose that `netdata.conf` is configured with the
378
-following labels:
377
+Defines the list of labels present on a host. See our [host labels tutorial](../docs/tutorials/using-host-labels.md) for
378
+an explanation of host labels and how to implement them.
379
+
380
+For example, let's suppose that `netdata.conf` is configured with the following labels:
381
382
```yaml
383
[host labels]
@@ -391,7 +393,7 @@ And more labels in `netdata.conf` for workstations:
393
room = workstation
394
```
395
394
-By defining labels inside of `netdata.conf`, you can now apply labels to alarms. For example, you can add the following
396
+By defining labels inside of `netdata.conf`, you can now apply labels to alarms. For example, you can add the following
397
line to any alarms you'd like to apply to hosts that have the label `room = server`.
398
399
```yaml