@cryptotaxi247 / netdata-1 / commits / 870683888

Add a plugin for Unbound DNS stats.

Unbound is a caching recursive DNS resolver often used to provide a local DNS cache on Linux systems or to provide DNS service on a router for a local network. It provides a wide variety of useful and interesting statistics through it's remote control interface that can be collected by tools like Netdata. This plugin makes a direct connection to the Unbound remote control interface to pull statistics from it. Currently, it tracks query rates (including cache hit and miss rates), info about the internal request processing list, rudimentary timing info (average and median) for recursive upstream queries, and optionally info about how many objects are in each of the various caches. Most of the settings for the local system can be auto-detected if the Unbound config file is parseable as YAML (it is technically YAML, but they allow tabs as indentation, and the default configs shipped by most distributions use tabs instead of spaces). This plugin is disabled by default because it requires some non-trivial changes to the Unbound configuration to configure the control interface. Supports both TLS encapsulated TCP connections, and local unix sockets.

Austin S. Hemmelgarn committed May 1, 2018 at 13:26 UTC 8706838889e216e49aa8415b861d283cee22fa62

5 files changed +338

conf.d/python.d.conf

@@ -75,5 +75,6 @@ nginx_log: no

75 # squid: yes

76 # springboot: yes

77 # tomcat: yes

78 +unbound: no

79 # varnish: yes

80 # web_log: yes

conf.d/python.d/unbound.conf new

+86

@@ -0,0 +1,86 @@

1 +# netdata python.d.plugin configuration for unbound

2 +#

3 +# This file is in YaML format. Generally the format is:

4 +#

5 +# name: value

6 +#

7 +# There are 2 sections:

8 +# - global variables

9 +# - one or more JOBS

10 +#

11 +# JOBS allow you to collect values from multiple sources.

12 +# Each source will have its own set of charts.

13 +#

14 +# JOB parameters have to be indented (using spaces only, example below).

15 +

16 +# ----------------------------------------------------------------------

17 +# Global Variables

18 +# These variables set the defaults for all JOBs, however each JOB

19 +# may define its own, overriding the defaults.

20 +

21 +# update_every sets the default data collection frequency.

22 +# If unset, the python.d.plugin default is used.

23 +# update_every: 1

24 +

25 +# priority controls the order of charts at the netdata dashboard.

26 +# Lower numbers move the charts towards the top of the page.

27 +# If unset, the default for python.d.plugin is used.

28 +# priority: 60000

29 +

30 +# retries sets the number of retries to be made in case of failures.

31 +# If unset, the default for python.d.plugin is used.

32 +# Attempts to restore the service are made once every update_everye

33 +# and only if the module has collected values in the past.

34 +# retries: 60

35 +

36 +# autodetection_retry sets the job re-check interval in seconds.

37 +# The job is not deleted if check fails.

38 +# Attempts to start the job are made once every autodetection_retry.

39 +# This feature is disabled by default.

40 +# autodetection_retry: 0

41 +

42 +# ----------------------------------------------------------------------

43 +# JOBS (data collection sources)

44 +#

45 +# The default JOBS share the same *name*. JOBS with the same name

46 +# are mutually exclusive. Only one of them will be allowed running at

47 +# any time. This allows autodetection to try several alternatives and

48 +# pick the one that works.

49 +#

50 +# Any number of jobs is supported.

51 +#

52 +# All python.d.plugin JOBS (for all its modules) support a set of

53 +# predefined parameters. These are:

54 +#

55 +# job_name:

56 +# name: myname # the JOB's name as it will appear at the

57 +# # dashboard (by default is the job_name)

58 +# # JOBs sharing a name are mutually exclusive

59 +# update_every: 1 # the JOB's data collection frequency

60 +# priority: 60000 # the JOB's order on the dashboard

61 +# retries: 60 # the JOB's number of restoration attempts

62 +# autodetection_retry: 0 # the JOB's re-check interval in seconds

63 +#

64 +# Additionally to the above, unbound also supports the following:

65 +#

66 +# host: localhost # The host to connect to.

67 +# port: 8953 # WHat port to use (defaults to 8953)

68 +# socket: /path/to/socket # A path to a UNIX socket to use instead

69 +# # of a TCP connection

70 +# ssl_key: /path/to/key # The keyfile to use for authentication

71 +# ssl_cert: /path/to/key # The certificate to use for authentication

72 +# extended: false @ Whether to collect extended stats or not

73 +#

74 +# In addition to the above, you can set the following to try and

75 +# auto-detect settings based on the unbound configuration:

76 +#

77 +# ubconf: /etc/unbound/unbound.conf

78 +#

79 +# Note that the SSL key and certificate need to be readable by the user

80 +# unbound runs as if you're using the regular control interface.

81 +# If you're using a UNIX socket, that has to be readable by the netdata user.

82 +

83 +# The following should work for most users if they have unbound configured

84 +# correctly.

85 +local:

86 + ubconf: /etc/unbound/unbound.conf

python.d/README.md

+71

@@ -2227,6 +2227,77 @@ Without configuration, module attempts to connect to `http://localhost:8080/heal

2227

2228 ---

2229

2230 +# Unbound

2231 +

2232 +Monitoring uses the remote control interface to fetch statistics.

2233 +

2234 +Provides the following charts:

2235 +

2236 +1. **Queries Processed**

2237 + * Ratelimited

2238 + * Cache Misses

2239 + * Cache Hits

2240 + * Expired

2241 + * Prefetched

2242 + * Recursive

2243 +

2244 +2. **Request List**

2245 + * Average Size

2246 + * Max Size

2247 + * Overwritten Requests

2248 + * Overruns

2249 + * Current Size

2250 + * User Requests

2251 +

2252 +3. **Recursion Timings**

2253 + * Average recursion processing time

2254 + * Median recursion processing time

2255 +

2256 +If extended stats are enabled, also provides:

2257 +

2258 +4. **Cache Sizes**

2259 + * Message Cache

2260 + * RRset Cache

2261 + * Infra Cache

2262 + * DNSSEC Key Cache

2263 + * DNSCrypt Shared Secret Cache

2264 + * DNSCrypt Nonce Cache

2265 +

2266 +### configuration

2267 +

2268 +Unbound must be manually configured to enable the remote-control protocol.

2269 +Check the Unbound documentation for info on how to do this. Additionally,

2270 +if you want to take advantage of the autodetection this plugin offers,

2271 +you will need to make sure your `unbound.conf` file only uses spaces for

2272 +indentation (the default config shipped by most distributions uses tabs

2273 +instead of spaces).

2274 +

2275 +Once you have the Unbound control protocol enabled, you need to make sure

2276 +that either the certificate and key are readable by Netdata (if you're

2277 +using the regular control interface), or that the socket is accessible

2278 +to Netdata (if you're using a UNIX socket for the contorl interface).

2279 +

2280 +By default, for the local system, every thing can be auto-detected

2281 +assumign Unbound is configured correctly and has been told to listen

2282 +on the loopback interface or a UNIX socket. This is done by looking

2283 +up info in the Unbound config file specified by the `ubconf` key.

2284 +

2285 +To enable extended stats for a given job, add `extended: yes` to the

2286 +definition.

2287 +

2288 +A basic local configuration with extended statistics looks like this:

2289 +

2290 +```yaml

2291 +local:

2292 + ubconf: /etc/unbound/unbound.conf

2293 + extended: yes

2294 +```

2295 +

2296 +While it's a bit more complicated to set up correctly, it is recommended

2297 +that you use a UNIX socket as it provides far better performance.

2298 +

2299 +---

2300 +

2301 # varnish cache

2302

2303 Module uses the `varnishstat` command to provide varnish cache statistics.

python.d/unbound.chart.py new

+158

@@ -0,0 +1,158 @@

1 +# -*- coding: utf-8 -*-

2 +# Description: unbound netdata python.d module

3 +# Author: Austin S. Hemmelgarn (Ferroin)

4 +

5 +import os

6 +import yaml

7 +

8 +from base.FrameworkServices.SocketService import SocketService

9 +

10 +

11 +ORDER = ['queries', 'reqlist', 'recursion']

12 +

13 +CHARTS = {

14 + 'queries': {

15 + 'options': [None, 'Queries Processed', 'queries', 'Unbound', 'unbound.queries', 'line'],

16 + 'lines': [

17 + ['ratelimit', 'Ratelimited', 'absolute', 1, 1],

18 + ['cachemiss', 'Cache Miss', 'absolute', 1, 1],

19 + ['cachehit', 'Cache Hit', 'absolute', 1, 1],

20 + ['expired', 'Expired', 'absolute', 1, 1],

21 + ['prefetch', 'Prefetched', 'absolute', 1, 1],

22 + ['recursive', 'Recursive', 'absolute', 1, 1]

23 + ]

24 + },

25 + 'reqlist': {

26 + 'options': [None, 'Request List', 'items', 'Unbound', 'unbound.reqlist', 'line'],

27 + 'lines': [

28 + ['reqlist_avg', 'Average Size', 'absolute', 1, 1],

29 + ['reqlist_max', 'Maximum Size', 'absolute', 1, 1],

30 + ['reqlist_overwritten', 'Overwritten Requests', 'absolute', 1, 1],

31 + ['reqlist_exceeded', 'Overruns', 'absolute', 1, 1],

32 + ['reqlist_current', 'Current Size', 'absolute', 1, 1],

33 + ['reqlist_user', 'User Requests', 'absolute', 1, 1]

34 + ]

35 + },

36 + 'recursion': {

37 + 'options': [None, 'Recursion Timings', 'seconds', 'Unbound', 'unbound.recursion', 'line'],

38 + 'lines': [

39 + ['recursive_avg', 'Average', 'absolute', 1, 1],

40 + ['recursive_med', 'Median', 'absolute', 1, 1]

41 + ]

42 + }

43 +}

44 +

45 +# These get added too if we are told to use extended stats.

46 +EXTENDED_ORDER = ['cache']

47 +

48 +EXTENDED_CHARTS = {

49 + 'cache': {

50 + 'options': [None, 'Cache Sizes', 'items', 'Unbound', 'unbound.cache', 'line'],

51 + 'lines': [

52 + ['cache_message', 'Message Cache', 'absolute', 1, 1],

53 + ['cache_rrset', 'RRSet Cache', 'absolute', 1, 1],

54 + ['cache_infra', 'Infra Cache', 'absolute', 1, 1],

55 + ['cache_key', 'DNSSEC Key Cache', 'absolute', 1, 1],

56 + ['cache_dnscss', 'DNSCrypt Shared Secret Cache', 'absolute', 1, 1],

57 + ['cache_dnscn', 'DNSCrypt Nonce Cache', 'absolute', 1, 1]

58 + ]

59 + }

60 +}

61 +

62 +# This maps the Unbound stat names to our names.

63 +STAT_MAP = {

64 + 'total.num.queries_ip_ratelimited': 'ratelimit',

65 + 'total.num.cachehits': 'cachehit',

66 + 'total.num.cachemiss': 'cachemiss',

67 + 'total.num.zero_ttl': 'expired',

68 + 'total.num.prefetch': 'prefetch',

69 + 'total.num.recursivereplies': 'recursive',

70 + 'total.requestlist.avg': 'reqlist_avg',

71 + 'total.requestlist.max': 'reqlist_max',

72 + 'total.requestlist.overwritten': 'reqlist_overwritten',

73 + 'total.requestlist.exceeded': 'reqlist_exceeded',

74 + 'total.requestlist.current.all': 'reqlist_current',

75 + 'total.requestlist.current.user': 'reqlist_user',

76 + 'total.recursion.time.avg': 'recursive_avg',

77 + 'total.recursion.time.median': 'recursive_med',

78 + 'msg.cache.count': 'cache_message',

79 + 'rrset.cache.count': 'cache_rrset',

80 + 'infra.cache.count': 'cache_infra',

81 + 'key.cache.count': 'cache_key',

82 + 'dnscrypt_shared_secret.cache.count': 'cache_dnscss',

83 + 'dnscrypt_nonce.cache.count': 'cache_dnscn'

84 +}

85 +

86 +

87 +class Service(SocketService):

88 + def __init__(self, configuration=None, name=None):

89 + # The unbound control protocol is always SSL encapsulated

90 + # unless it's used over a UNIX socket, so enable SSL _before_

91 + # doing the normal SocketService initialization.

92 + configuration['ssl'] = True

93 + self.port = 8935

94 + SocketService.__init__(self, configuration, name)

95 + self.ext = self.configuration.get('extended', None)

96 + self.ubconf = self.configuration.get('ubconf', '/etc/unbound/unbound.conf')

97 + self.order = ORDER

98 + self.definitions = CHARTS

99 + if self.ext:

100 + self.order = self.order + EXTENDED_ORDER

101 + self.definitions.update(EXTENDED_CHARTS)

102 + self.request = 'UBCT1 stats\n'

103 + self._parse_config()

104 + self.debug('Unbound config: {0}'.format(self.ubconf)

105 + if os.access(self.ubconf, os.R_OK):

106 + with open(self.ubconf, 'r') as ubconf:

107 + try:

108 + conf = yaml.load(ubconf)

109 + except yaml.YAMLError:

110 + conf = dict()

111 + if self.ext is None:

112 + if 'extended-statistics' in conf['server'].keys():

113 + self.ext = conf['server']['extended-statistics']

114 + if 'remote-control' in conf.keys():

115 + if conf['remote-control'].get('control-use-cert', False):

116 + if not self.key:

117 + self.key = conf['remote-control'].get('control-key-file', '/etc/unbound/unbound_control.key')

118 + if not self.cert:

119 + self.cert = conf['remote-control'].get('control-cert-file', '/etc/unbound/unbound_control.pem')

120 + self.port = conf['remote-control'].get('control-port', 8953)

121 + else:

122 + if not self.unix_socket:

123 + self.unix_socket = conf['remote-control'].get('control-interface')

124 + self.debug('Extended stats: {0}'.format(self.ext))

125 + if self.unix_socket:

126 + self.debug('Using unix socket: {0}'.format(self.unix_socket))

127 + for key in self.definitions.keys():

128 + self.definitions[key]['options'][4] = 'Local'

129 + else:

130 + self.debug('Connecting to: {0}:{1}'.format(self.host, self.port))

131 + self.debug('Using key: {0}'.format(self.key))

132 + self.debug('Using certificate: {0}'.format(self.cert))

133 + for key in self.definitions.keys():

134 + self.definitions[key]['options'][4] = self.host

135 +

136 +

137 + def check(self):

138 + # We need to check that auth works, otherwise there's no point.

139 + self._connect()

140 + self._disconnect()

141 + return bool(self._sock)

142 +

143 + def _check_raw_data(self, data):

144 + # The server will close the connection when it's done sending

145 + # data, so just keep looping until that happens.

146 + return False

147 +

148 + def _get_data(self):

149 + raw = self._get_raw_data()

150 + data = dict()

151 + tmp = dict()

152 + for line in raw.splitlines():

153 + stat = line.split('=')

154 + tmp[stat[0]] = stat[1]

155 + for item in STAT_MAP.keys():

156 + if item in tmp.keys():

157 + data[STAT_MAP[item]] = float(tmp[item])

158 + return data

web/dashboard_info.js

+22

@@ -359,6 +359,12 @@ netdataDashboard.menu = {

359 title: 'ntpd',

360 icon: '',

361 info: 'Provides statistics for the internal variables of the Network Time Protocol daemon <a href="http://www.ntp.org/">ntpd</a> and optional including the configured peers (if enabled in the module configuration). The module presents the performance metrics as shown by <a href="http://doc.ntp.org/current-stable/ntpq.html">ntpq</a> (the standard NTP query program) using NTP mode 6 UDP packets to communicate with the NTP server.'

362 + },

363 +

364 + 'unbound' {

365 + title: 'Unbound',

366 + icon: '',

367 + info: undefined

368 }

369 };

370

@@ -2064,6 +2070,22 @@ netdataDashboard.context = {

2070

2071 'ntpd.peer_precision': {

2072 height: 0.2

2073 + },

2074 +

2075 + 'unbound.queries': {

2076 + info: 'Shows the number of queries being processed of each type. Note that <code>Recursive</code> queries are also accounted as cache misses.'

2077 + },

2078 +

2079 + 'unbound.reqlist': {

2080 + info: 'Shows various stats about Unbound\'s internal request list.'

2081 + },

2082 +

2083 + 'unbound.recursion': {

2084 + info: 'Average and median time to complete recursive name resolution.'

2085 + },

2086 +

2087 + 'unbound.cache': {

2088 + info: 'The number of items in each of the various caches.'

2089 }

2090

2091 // ------------------------------------------------------------------------