@cryptotaxi247 / netdata-1 / commits / dd36c217d

Add a python.d module for monitoring BOINC clients.

This adds a python.d module for monitoring Berkeley Open Infrastructure Network Computing clients. This is the framework utilized by a vast majority of publicly run distributed computing projects (including SETI@Home and World Community Grid). This only tracks the number of tasks in various task states on the system, as almost everything else is trivial to track using other existing plugins (such as apps.plugin). It utilizes the same RPC mechanism that the official GUI management tool for BOINC clients uses. In most cases, the data this tracks is not hugely interesting (it doesn't change very often, and generally doesn't change very much on any given system), but there are a handful of situations that it lets us provide alerts for that are of particular interest to users running BOINC on headless systems (namely computational errors, reporting failures, and running out of tasks to run). Internally, BOINC has 3 state-machines that a given task will advance through, the task state, scheduling state, and process state. The task state tracks the high-level status of the task (is it new, ready to run, finished running and being reported, hit a computational error, etc). the scheduling state tracks whether the scheduling algorithm says the task can run or not, and the process state tracks the status of the aassociated process for the task that does the actual computing. This module provides charts tracking task counts in each state for each of these state machines, as well as one counting total number of local tasks, and how many tasks are 'active' (that is, how many are being considered for processing). We also provide a set of default alarms to alert on compute errors, upload/reporting failures, an empty local task queue, and a low number of active tasks. Currently, the module only runs on Linux systems because it relys on a Linux specific hack to detect a disconnected socket, but it can monitor BOINC clients on any platform remotely (including Windows systems).

Austin S. Hemmelgarn committed Jun 5, 2018 at 09:23 UTC dd36c217d45cc8992df3d8a316fa30bb0dfe9952

7 files changed +362

conf.d/health.d/boinc.conf new

+62

@@ -0,0 +1,62 @@

1 +# Alarms for various BOINC issues.

2 +

3 +# Warn on any compute errors encountered.

4 +template: boinc_compute_errors

5 + on: boinc.states

6 + os: *

7 + hosts: *

8 +families: *

9 + lookup: sum -10m unaligned of comperror

10 + units: tasks

11 + every: 1m

12 + warn: $this > 1

13 + crit: $this > 4

14 + delay: up 1m down 5m multiplier 1.5 max 1h

15 + info: the total number of compute errors over the past 10 minutes

16 + to: sysadmin

17 +

18 +# Warn on lots of upload errors

19 +template: boinc_upload_errors

20 + on: boinc.states

21 + os: *

22 + hosts: *

23 +families: *

24 + lookup: max -10m unaligned of upload_failed

25 + units: tasks

26 + every: 1m

27 + warn: $this > 2

28 + crit: $this > 4

29 + delay: up 1m down 5m multiplier 1.5 max 1h

30 + info: the average number of failed uploads over the past 10 minutes

31 + to: sysadmin

32 +

33 +# Warn on the task queue being empty

34 +template: boinc_total_tasks

35 + on: boinc.tasks

36 + os: *

37 + hosts: *

38 +families: *

39 + lookup: average -10m unaligned of total

40 + units: tasks

41 + every: 1m

42 + warn: $this < 1

43 + crit: $this < 0.1

44 + delay: up 5m down 10m multiplier 1.5 max 1h

45 + info: the total number of locally available tasks

46 + to: sysadmin

47 +

48 +# Warn on no active tasks with a non-empty queue

49 +template: boinc_active_tasks

50 + on: boinc.tasks

51 + os: *

52 + hosts: *

53 +families: *

54 + lookup: average -10m unaligned of active

55 + calc: ($boinc_total_tasks >= 1) ? ($this) : (inf)

56 + units: tasks

57 + every: 1m

58 + warn: $this < 1

59 + crit: $this < 0.1

60 + delay: up 5m down 10m multiplier 1.5 max 1h

61 + info: the total number of active tasks

62 + to: sysadmin

conf.d/python.d.conf

@@ -24,6 +24,7 @@ apache_cache: no

24 # apache: yes

25 # beanstalk: yes

26 # bind_rndc: yes

27 +# boinc: yes

28 # ceph: yes

29 chrony: no

30 # couchdb: yes

conf.d/python.d/boinc.conf new

+68

@@ -0,0 +1,68 @@

1 +# netdata python.d.plugin configuration for boinc

2 +#

3 +# This file is in YaML format. Generally the format is:

4 +#

5 +# name: value

6 +#

7 +# There are 2 sections:

8 +# - global variables

9 +# - one or more JOBS

10 +#

11 +# JOBS allow you to collect values from multiple sources.

12 +# Each source will have its own set of charts.

13 +#

14 +# JOB parameters have to be indented (using spaces only, example below).

15 +

16 +# ----------------------------------------------------------------------

17 +# Global Variables

18 +# These variables set the defaults for all JOBs, however each JOB

19 +# may define its own, overriding the defaults.

20 +

21 +# update_every sets the default data collection frequency.

22 +# If unset, the python.d.plugin default is used.

23 +# update_every: 1

24 +

25 +# priority controls the order of charts at the netdata dashboard.

26 +# Lower numbers move the charts towards the top of the page.

27 +# If unset, the default for python.d.plugin is used.

28 +# priority: 60000

29 +

30 +# retries sets the number of retries to be made in case of failures.

31 +# If unset, the default for python.d.plugin is used.

32 +# Attempts to restore the service are made once every update_every

33 +# and only if the module has collected values in the past.

34 +# retries: 60

35 +

36 +# autodetection_retry sets the job re-check interval in seconds.

37 +# The job is not deleted if check fails.

38 +# Attempts to start the job are made once every autodetection_retry.

39 +# This feature is disabled by default.

40 +# autodetection_retry: 0

41 +

42 +# ----------------------------------------------------------------------

43 +# JOBS (data collection sources)

44 +#

45 +# The default JOBS share the same *name*. JOBS with the same name

46 +# are mutually exclusive. Only one of them will be allowed running at

47 +# any time. This allows autodetection to try several alternatives and

48 +# pick the one that works.

49 +#

50 +# Any number of jobs is supported.

51 +#

52 +# All python.d.plugin JOBS (for all its modules) support a set of

53 +# predefined parameters. These are:

54 +#

55 +# job_name:

56 +# name: myname # the JOB's name as it will appear at the

57 +# # dashboard (by default is the job_name)

58 +# # JOBs sharing a name are mutually exclusive

59 +# update_every: 1 # the JOB's data collection frequency

60 +# priority: 60000 # the JOB's order on the dashboard

61 +# retries: 60 # the JOB's number of restoration attempts

62 +# autodetection_retry: 0 # the JOB's re-check interval in seconds

63 +#

64 +# Additionally to the above, boinc also supports the following:

65 +#

66 +# hostname: localhost # The host running the BOINC client

67 +# port: 31416 # The remote GUI RPC port for BOINC

68 +# password: '' # The remote GUI RPC password

python.d/Makefile.am

@@ -17,6 +17,7 @@ dist_python_DATA = \

17 apache.chart.py \

18 beanstalk.chart.py \

19 bind_rndc.chart.py \

20 + boinc.chart.py \

21 ceph.chart.py \

22 chrony.chart.py \

23 couchdb.chart.py \

python.d/README.md

+29

@@ -291,6 +291,35 @@ If no configuration is given, module will attempt to read named.stats file at `

291

292 ---

293

294 +# boinc

295 +

296 +This module monitors task counts for the Berkely Open Infrastructure

297 +Networking Computing (BOINC) distributed computing client using the same

298 +RPC interface that the BOINC monitoring GUI does.

299 +

300 +It provides charts tracking the total number of tasks and active tasks,

301 +as well as ones tracking each of the possible states for tasks.

302 +

303 +### configuration

304 +

305 +BOINC requires use of a password to access it's RPC interface. You can

306 +find this password in the `gui_rpc_auth.cfg` file in your BOINC directory.

307 +

308 +By default, the module will try to auto-detect the password by looking

309 +in `/var/lib/boinc` for this file (this is the location most Linux

310 +distributions use for a system-wide BOINC installation), so things may

311 +just work without needing configuration for the local system.

312 +

313 +You can monitor remote systems as well:

314 +

315 +```yaml

316 +remote:

317 + hostname: some-host

318 + password: some-password

319 +```

320 +

321 +---

322 +

323 # chrony

324

325 This module monitors the precision and statistics of a local chronyd server.

python.d/boinc.chart.py new

+178

@@ -0,0 +1,178 @@

1 +# -*- coding: utf-8 -*-

2 +# Description: BOINC netdata python.d module

3 +# Author: Austin S. Hemmelgarn (Ferroin)

4 +# SPDX-License-Identifier: GPL-3.0+

5 +

6 +import platform

7 +import socket

8 +

9 +from copy import deepcopy

10 +

11 +from bases.FrameworkServices.SimpleService import SimpleService

12 +

13 +from third_party import boinc_client

14 +

15 +

16 +ORDER = ['tasks', 'states', 'sched_states', 'process_states']

17 +

18 +CHARTS = {

19 + 'tasks': {

20 + 'options': [None, 'Overall Tasks', 'tasks', 'boinc', 'boinc.tasks', 'line'],

21 + 'lines': [

22 + ['total', 'Total', 'absolute', 1, 1],

23 + ['active', 'Active', 'absolute', 1, 1]

24 + ]

25 + },

26 + 'states': {

27 + 'options': [None, 'Tasks per State', 'tasks', 'boinc', 'boinc.states', 'line'],

28 + 'lines': [

29 + ['new', 'New', 'absolute', 1, 1],

30 + ['downloading', 'Downloading', 'absolute', 1, 1],

31 + ['downloaded', 'Ready to Run', 'absolute', 1, 1],

32 + ['comperror', 'Compute Errors', 'absolute', 1, 1],

33 + ['uploading', 'Uploading', 'absolute', 1, 1],

34 + ['uploaded', 'Uploaded', 'absolute', 1, 1],

35 + ['aborted', 'Aborted', 'absolute', 1, 1],

36 + ['upload_failed', 'Failed Uploads', 'absolute', 1, 1]

37 + ]

38 + },

39 + 'sched_states': {

40 + 'options': [None, 'Tasks per Scheduler State', 'tasks', 'boinc', 'boinc.sched', 'line'],

41 + 'lines': [

42 + ['uninit_sched', 'Uninitialized', 'absolute', 1, 1],

43 + ['preempted', 'Preempted', 'absolute', 1, 1],

44 + ['scheduled', 'Scheduled', 'absolute', 1, 1]

45 + ]

46 + },

47 + 'process_states': {

48 + 'options': [None, 'Tasks per Process State', 'tasks', 'boinc', 'boinc.process', 'line'],

49 + 'lines': [

50 + ['uninit_proc', 'Uninitialized', 'absolute', 1, 1],

51 + ['executing', 'Executing', 'absolute', 1, 1],

52 + ['suspended', 'Suspended', 'absolute', 1, 1],

53 + ['aborting', 'Aborted', 'absolute', 1, 1],

54 + ['quit', 'Quit', 'absolute', 1, 1],

55 + ['copy_pending', 'Copy Pending', 'absolute', 1, 1]

56 + ]

57 + }

58 +}

59 +

60 +# A simple template used for pre-loading the return dictionary to make

61 +# the _get_data() method simpler.

62 +_DATA_TEMPLATE = {

63 + 'total': 0,

64 + 'active': 0,

65 + 'new': 0,

66 + 'downloading': 0,

67 + 'downloaded': 0,

68 + 'comperror': 0,

69 + 'uploading': 0,

70 + 'uploaded': 0,

71 + 'aborted': 0,

72 + 'upload_failed': 0,

73 + 'uninit_sched': 0,

74 + 'preempted': 0,

75 + 'scheduled': 0,

76 + 'uninit_proc': 0,

77 + 'executing': 0,

78 + 'suspended': 0,

79 + 'aborting': 0,

80 + 'quit': 0,

81 + 'copy_pending': 0

82 +}

83 +

84 +# Map task states to dimensions

85 +_TASK_MAP = {

86 + boinc_client.ResultState.NEW: 'new',

87 + boinc_client.ResultState.FILES_DOWNLOADING: 'downloading',

88 + boinc_client.ResultState.FILES_DOWNLOADED: 'downloaded',

89 + boinc_client.ResultState.COMPUTE_ERROR: 'comperror',

90 + boinc_client.ResultState.FILES_UPLOADING: 'uploading',

91 + boinc_client.ResultState.FILES_UPLOADED: 'uploaded',

92 + boinc_client.ResultState.ABORTED: 'aborted',

93 + boinc_client.ResultState.UPLOAD_FAILED: 'upload_failed'

94 +}

95 +

96 +# Map scheduler states to dimensions

97 +_SCHED_MAP = {

98 + boinc_client.CpuSched.UNINITIALIZED: 'uninit_sched',

99 + boinc_client.CpuSched.PREEMPTED: 'preempted',

100 + boinc_client.CpuSched.SCHEDULED: 'scheduled',

101 +}

102 +

103 +# Maps process states to dimensions

104 +_PROC_MAP = {

105 + boinc_client.Process.UNINITIALIZED: 'uninit_proc',

106 + boinc_client.Process.EXECUTING: 'executing',

107 + boinc_client.Process.SUSPENDED: 'suspended',

108 + boinc_client.Process.ABORT_PENDING: 'aborted',

109 + boinc_client.Process.QUIT_PENDING: 'quit',

110 + boinc_client.Process.COPY_PENDING: 'copy_pending'

111 +}

112 +

113 +

114 +class Service(SimpleService):

115 + def __init__(self, configuration=None, name=None):

116 + SimpleService.__init__(self, configuration=configuration, name=name)

117 + self.order = ORDER

118 + self.definitions = CHARTS

119 + self.host = self.configuration.get('host', 'localhost')

120 + self.port = self.configuration.get('port', 0)

121 + self.password = self.configuration.get('password', '')

122 + self.client = boinc_client.BoincClient(host=self.host, port=self.port, passwd=self.password)

123 + self.alive = False

124 +

125 + def check(self):

126 + if platform.system() != 'Linux':

127 + self.error('Only supported on Linux.')

128 + return False

129 + self.connect()

130 + if self.client.connected and self.client.authorized:

131 + self.alive = True

132 + else:

133 + self.alive = False

134 + return self.alive

135 +

136 + def connect(self):

137 + self.client.connect()

138 +

139 + def reconnect(self):

140 + try:

141 + self.client.disconnect()

142 + except socket.error:

143 + pass

144 + self.client.connect()

145 + if self.client.connected and self.client.authorized:

146 + self.alive = True

147 + else:

148 + self.alive = False

149 + return self.alive

150 +

151 + def is_alive(self):

152 + if (not self.alive) or \

153 + self.client.rpc.sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_INFO, 0) != 1:

154 + return self.reconnect()

155 + return True

156 +

157 + def _get_data(self):

158 + if not self.is_alive():

159 + return None

160 + data = deepcopy(_DATA_TEMPLATE)

161 + results = []

162 + try:

163 + results = self.client.get_tasks()

164 + except socket.error:

165 + self.error('Connection is dead')

166 + self.alive = False

167 + return None

168 + for task in results:

169 + data['total'] += 1

170 + data[_TASK_MAP[task.state]] += 1

171 + try:

172 + if task.active_task:

173 + data['active'] += 1

174 + data[_SCHED_MAP[task.scheduler_state]] += 1

175 + data[_PROC_MAP[task.active_task_state]] += 1

176 + except AttributeError:

177 + pass

178 + return data

web/dashboard_info.js

+23

@@ -373,10 +373,17 @@ netdataDashboard.menu = {

373 icon: '',

374 info: 'Provides basic performance statistics for the <a href="https://www.spigotmc.org/">Spigot Minecraft</a> server.'

375 },

376 +

377 'unbound': {

378 title: 'Unbound',

379 icon: '',

380 info: undefined

381 + },

382 +

383 + 'boinc': {

384 + title: 'BOINC'

385 + icon: '',

386 + info: 'Provides task counts for <a href="http://boinc.berkeley.edu/">BOINC</a> distributed computing clients.'

387 }

388 };

389

@@ -2119,6 +2126,22 @@ netdataDashboard.context = {

2126

2127 'unbound.threads.recursion': {

2128 height: 0.2

2129 + },

2130 +

2131 + 'boinc.tasks': {

2132 + info: 'The total number of tasks and the number of active tasks. Active tasks are those which are either currently being processed, or are partialy processed but suspended.

2133 + },

2134 +

2135 + 'boinc.states': {

2136 + info: 'Counts of tasks in each task state. The normal sequence of states is <code>New</code>, <code>Downloading</code>, <code>Ready to Run<c/ode>, <code>Uploading</code>, <code>Uploaded</code>. Tasks which are marked <code>Ready to Run</code> may be actively running, or may be waiting to be scheduled. <code>Compute Errors</code> are tasks which failed for some reason during execution. <code>Aborted</code> tasks were manually cancelled, and will not be processed. <code>Failed Uploads</code> are otherwise finished tasks which failed to upload to the server, and usually indicate networking issues.'

2137 + },

2138 +

2139 + 'boinc.sched': {

2140 + info: 'Counts of active tasks in each scheduling state. <code>Scheduled</code> tasks are the ones which will run if the system is permitted to process tasks. <code>Preempted</code> tasks are on standby, and will run if a <code>Scheduled</code> task stops running for some reason. <code>Uninitialized</code> tasks should never be present, and indicate tha the scheduler has not tried to schedule them yet.'

2141 + },

2142 +

2143 + 'boinc.process': {

2144 + info: 'Counts of active tasks in each process state. <code>Executing</code> tasks are running right now. <code>Suspended</code> tasks have an associated process, but are not currently running (either because the system isn\'t processing any tasks right now, or because they have been preempted by higher priority tasks). <code>Quit</code> tasks are exiting gracefully. <code>Aborted</code> tasks exceeded some resource limit, and are being shut down. code>Copy Pending</code> tasks are waiting on a background file transfer to finish. <code>Uninitialized</code> tasks do not have an associated process yet.'

2145 }

2146

2147 // ------------------------------------------------------------------------