Tutorials to support v1.19 release (#7359)
* Initialize two 1.19 tutorials * Continue work on tutorials * Working on weblog tutorial * Finished very rough drafts of collector tutorials * Moved filenames to be more descriptive/SEO-friendly * Fixes for Ilya and Austin * Fixes for Ilya * Fix link in tutorial
Joel Hans committed
Nov 27, 2019 at 20:31 UTC
bfd54ca6e511f3cbb3a406125f66e36657ec58d6
2 files changed
+287
docs/tutorials/collect-apache-nginx-web-logs.md
new
+155
@@ -0,0 +1,155 @@
1
+# Monitor Nginx or Apache web server log files with Netdata
2
+
3
+Log files have been a critical resource for developers and system administrators who want to understand the health and
4
+performance of their web servers, and Netdata is taking important steps to make them even more valuable.
5
+
6
+By parsing web server log files with Netdata, and seeing the volume of redirects, requests, or server errors over time,
7
+you can better understand what's happening on your infrastructure. Too many bad requests? Maybe a recent deploy missed a
8
+few small SVG icons. Too many requsests? Time to batten down the hatches—it's a DDoS.
9
+
10
+Netdata has been capable of monitoring web log files for quite some time, thanks for the [weblog python.d
11
+module](../../collectors/python.d.plugin/web_log/README.md), but we recently refactored this module in Go, and that
12
+effort comes with a ton of improvements.
13
+
14
+You can now use the [LTSV log format](http://ltsv.org/), track TLS and cipher usage, and the whole parser is faster than
15
+ever. In one test on a system with SSD storage, the collector consistently parsed the logs for 200,000 requests in
16
+200ms, using ~30% of a single core. To learn more about these improvements, see our [v1.19 release post](https://blog.netdata.cloud/posts/release-1.19/).
17
+
18
+The [go.d plugin](https://github.com/netdata/go.d.plugin/tree/master/modules/weblog) is currently compatible with
19
+[Nginx](https://nginx.org/en/) and [Apache](https://httpd.apache.org/).
20
+
21
+This tutorial will walk you through using the new Go-based web log collector to turn the logs these web servers
22
+constantly write to into real-time insights into your infrastructure.
23
+
24
+## Set up your web servers
25
+
26
+As with all data sources, Netdata can auto-detect Nginx or Apache servers if you installed them using their standard
27
+installation procedures.
28
+
29
+Almost all web server installations will need _no_ configuration to start collecting metrics. As long as your web server
30
+has readable access log file, you can configure the web log plugin to access and parse it.
31
+
32
+## Configure the web log collector
33
+
34
+To use the Go version of this plugin, you need to explicitly enable it, and disable the depreciated Python version.
35
+First, open `python.d.conf`:
36
+
37
+```bash
38
+cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/
39
+./edit-config python.d.conf
40
+```
41
+
42
+Find the `web_log` line, uncomment it, and set it to `web_log: no`. Next, open the `go.d.conf` file for editing.
43
+
44
+```bash
45
+./edit-config go.d.conf
46
+```
47
+
48
+Find the `web_log` line again, uncomment it, and set it to `web_log: yes`.
49
+
50
+Finally, restart Netdata with `service netdata restart`, or the appropriate method for your system. You should see
51
+metrics in your Netdata dashboard!
52
+
53
+
55
+
56
+If you don't see web log charts, or **web log nginx**/**web log apache** menus on the right-hand side of your dashboard,
57
+continue reading for other configuration options.
58
+
59
+## Custom configuration of the web log collector
60
+
61
+The web log collector's default configuration comes with a few example jobs that should cover most Linux distributions
62
+and their default locations for log files:
63
+
64
+```yaml
65
+# [ JOBS ]
66
+jobs:
67
+# NGINX
68
+# debian, arch
69
+ - name: nginx
70
+ path: /var/log/nginx/access.log
71
+
72
+# gentoo
73
+ - name: nginx
74
+ path: /var/log/nginx/localhost.access_log
75
+
76
+# APACHE
77
+# debian
78
+ - name: apache
79
+ path: /var/log/apache2/access.log
80
+
81
+# gentoo
82
+ - name: apache
83
+ path: /var/log/apache2/access_log
84
+
85
+# arch
86
+ - name: apache
87
+ path: /var/log/httpd/access_log
88
+
89
+# debian
90
+ - name: apache_vhosts
91
+ path: /var/log/apache2/other_vhosts_access.log
92
+
93
+# GUNICORN
94
+ - name: gunicorn
95
+ path: /var/log/gunicorn/access.log
96
+
97
+ - name: gunicorn
98
+ path: /var/log/gunicorn/gunicorn-access.log
99
+```
100
+
101
+However, if your log files were not auto-detected, it might be because they are in a different location. Try the default
102
+`weblog.conf` file.
103
+
104
+```bash
105
+./edit-config go.d/weblog.conf
106
+```
107
+
108
+To create a new custom configuration, you need to set the `path` parameter to point to your web server's access log
109
+file. You can give it a `name` as well, and set the `log_type` to `auto`.
110
+
111
+```yaml
112
+jobs:
113
+ - name: example
114
+ path: /path/to/file.log
115
+ log_type: auto
116
+```
117
+
118
+Restart Netdata with `service netdata restart` or the appropriate method for your system. Netdata should pick up your
119
+web server's access log and begin showing real-time charts!
120
+
121
+### Custom log formats and fields
122
+
123
+The web log collector is capable of parsing custom Nginx and Apache log formats and presenting them as charts, but we'll
124
+leave that topic for a separate tutorial.
125
+
126
+We do have [extensive documentation](../../collectors/go.d.plugin/modules/weblog/#custom-log-format) on how to build
127
+custom parsing for Nginx and Apache logs.
128
+
129
+## Tweak web log collector alarms
130
+
131
+Over time, we've created some default alarms for web log monitoring. These alarms are designed to work only when your
132
+web server is receiving more than 120 requests per minute. Otherwise, there's simply not enough data to make conclusions
133
+about what is "too few" or "too many."
134
+
135
+- [web log alarms](https://raw.githubusercontent.com/netdata/netdata/master/health/health.d/web_log.conf).
136
+
137
+You can also edit this file directly with `edit-config`:
138
+
139
+```bash
140
+./edit-config health.d/weblog.conf
141
+```
142
+
143
+For more information about editing the defaults or writing new alarm entities, see our [health monitoring
144
+documentation](../../health/README.md).
145
+
146
+## What's next?
147
+
148
+Now that you have web log collection up and running, we recommend you take a look at the documentation for our
149
+[python.d](../../collectors/python.d.plugin/web_log/README.md) for some ideas of how you can turn these rather "boring"
150
+logs into powerful real-time tools for keeping your servers happy.
151
+
152
+Don't forget to give GitHub user [Wing924](https://github.com/Wing924) a big 👍 for his hard work in starting up the Go
153
+refactoring effort.
154
+
155
+[](<>)
docs/tutorials/collect-unbound-metrics.md
new
+132
@@ -0,0 +1,132 @@
1
+# Monitor Unbound DNS servers with Netdata
2
+
3
+[Unbound](https://nlnetlabs.nl/projects/unbound/about/) is a "validating, recursive, caching DNS resolver" from NLNet
4
+Labs. In v1.19 of Netdata, we release a completely refactored collector for collecting real-time metrics from Unbound
5
+servers and displaying them in Netdata dashboards.
6
+
7
+Unbound runs on FreeBSD, OpenBSD, NetBSD, MacOS, Linux, and Windows, and supports DNS-over-TLS, which ensures that DNS
8
+queries and answers are all encrypted with TLS. In theory, that should reduce the risk of eavesdropping or
9
+man-in-the-middle attacks when communicating to DNS servers.
10
+
11
+This tutorial will show you how to collect dozens of essential metrics from your Unbound servers with minimal
12
+configuration.
13
+
14
+## Set up your Unbound installation
15
+
16
+As with all data sources, Netdata can auto-detect Unbound servers if you installed them using the standard installation
17
+procedure.
18
+
19
+Regardless of whether you're connecting to a local or remote Unbound server, you need to be able to access the server's
20
+`remote-control` interface via an IP address, FQDN, or Unix socket.
21
+
22
+To set up the `remote-control` interface, you can use `unbound-control`. First, run `unbound-control-setup` to generate
23
+the TLS key files that will encrypt connections to the remote interface. Then add the following to the end of your
24
+`unbound.conf` configuration file. See the [Unbound
25
+documentation](https://nlnetlabs.nl/documentation/unbound/howto-setup/#setup-remote-control) for more details on using
26
+`unbound-control`, such as how to handle situations when Unbound is run under a unique user.
27
+
28
+```conf
29
+# enable remote-control
30
+remote-control:
31
+ control-enable: yes
32
+```
33
+
34
+Next, make your `unbound.conf`, `unbound_control.key`, and `unbound_control.pem` files readable by Netdata using [access
35
+control lists](https://wiki.archlinux.org/index.php/Access_Control_Lists) (ACL).
36
+
37
+```bash
38
+sudo setfacl -m user:netdata:r unbound.conf
39
+sudo setfacl -m user:netdata:r unbound_control.key
40
+sudo setfacl -m user:netdata:r unbound_control.pem
41
+```
42
+
43
+Finally, take note whether you're using Unbound in _cumulative_ or _non-cumulative_ mode. This will become relevant when
44
+configuring the collector.
45
+
46
+## Configure the Unbound collector
47
+
48
+You may not need to do any more configuration to have Netdata collect your Unbound metrics.
49
+
50
+If you followed the steps above to enable `remote-control` and make your Unbound files readable by Netdata, that should
51
+be enough. Restart Netdata with `service netdata restart`, or the appropriate method for your system. You should see
52
+Unbound metrics in your Netdata dashboard!
53
+
54
+
55
+
56
+If that failed, you will need to manually configure `unbound.conf`. See the next section for details.
57
+
58
+### Manual setup for a local Unbound server
59
+
60
+To configure Netdata's Unbound collector module, navigate to your Netdata configuration directory (typically at
61
+`/etc/netdata/`) and use `edit-config` to initialize and edit your Unbound configuration file.
62
+
63
+```bash
64
+cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/
65
+sudo ./edit-config go.d/unbound.conf
66
+```
67
+
68
+The file contains all the global and job-related parameters. The `name` setting is required, and two Unbound servers
69
+can't have the same name.
70
+
71
+> It is important you know whether your Unbound server is running in cumulative or non-cumulative mode, as a conflict
72
+> between modes will create incorrect charts.
73
+
74
+Here are two examples for local Unbound servers, which may work based on your unique setup:
75
+
76
+```yaml
77
+jobs:
78
+ - name: local
79
+ address: 127.0.0.1:8953
80
+ cumulative: no
81
+ use_tls: yes
82
+ tls_skip_verify: yes
83
+ tls_cert: /path/to/unbound_control.pem
84
+ tls_key: /path/to/unbound_control.key
85
+
86
+ - name: local
87
+ address: 127.0.0.1:8953
88
+ cumulative: yes
89
+ use_tls: no
90
+```
91
+
92
+Netdata will attempt to read `unbound.conf` to get the appropriate `address`, `cumulative`, `use_tls`, `tls_cert`, and
93
+`tls_key` parameters.
94
+
95
+Restart Netdata with `service netdata restart`, or the appropriate method for your system.
96
+
97
+### Manual setup for a remote Unbound server
98
+
99
+Collecting metrics from remote Unbound servers requires manual configuration. There are too many possibilities to cover
100
+all remote connections here, but the [default `unbound.conf`
101
+file](https://github.com/netdata/go.d.plugin/blob/master/config/go.d/unbound.conf) contains a few useful examples:
102
+
103
+```yaml
104
+jobs:
105
+ - name: remote
106
+ address: 203.0.113.10:8953
107
+ use_tls: no
108
+
109
+ - name: remote_cumulative
110
+ address: 203.0.113.11:8953
111
+ use_tls: no
112
+ cumulative: yes
113
+
114
+ - name: remote
115
+ address: 203.0.113.10:8953
116
+ cumulative: yes
117
+ use_tls: yes
118
+ tls_cert: /etc/unbound/unbound_control.pem
119
+ tls_key: /etc/unbound/unbound_control.key
120
+```
121
+
122
+To see all the available options, see the default [unbound.conf
123
+file](https://github.com/netdata/go.d.plugin/blob/master/config/go.d/unbound.conf).
124
+
125
+## What's next?
126
+
127
+Now that you're collecting metrics from your Unbound servers, let us know how it's working for you! There's always room
128
+for improvement or refinement based on real-world use cases. Feel free to [file an
129
+issue](https://github.com/netdata/netdata/issues/new?labels=bug%2C+needs+triage&template=bug_report.md) with your
130
+thoughts.
131
+
132
+[](<>)