@cryptotaxi247 / netdata-1 / commits / 0840d52e7

smartd_log refactor plus SCSI support (#4523)

* smartd_log: refactor plus scsi support init * smartd_log: ata attr class fix * smartd_log: whitespace * smartd_log: add __hash__ to BaseDisk * smartd_log: ata temp fix * smartd_log: ata power on hours fix * smartd_log: add ata 194 decode func * smartd_log: readme update * smartd_log: handle log rotation * smartd_log: add ata 1, 7, 206 attrs * smartd_log: minor * smartd_log: readme update

Ilya Mashchenko committed Nov 6, 2018 at 22:42 UTC 0840d52e7147902d7dfd417ca33a415c3da9a5c4
3 files changed +719 -306
collectors/python.d.plugin/smartd_log/README.md
+76 -13
@@ -2,29 +2,92 @@
2
3 Module monitor `smartd` log files to collect HDD/SSD S.M.A.R.T attributes.
4
5 -It produces following charts (you can add additional attributes in the module configuration file):
5 +**Requirements:**
6 +* `smartmontools`
7
7 -1. **Read Error Rate** attribute 1
8 +It produces following charts for SCSI devices:
9
9 -2. **Start/Stop Count** attribute 4
10 +1. **Read Error Corrected**
11
11 -3. **Reallocated Sectors Count** attribute 5
12 +2. **Read Error Uncorrected**
13
13 -4. **Seek Error Rate** attribute 7
14 +3. **Write Error Corrected**
15
15 -5. **Power-On Hours Count** attribute 9
16 +4. **Write Error Uncorrected**
17
17 -6. **Power Cycle Count** attribute 12
18 +5. **Verify Error Corrected**
19
19 -7. **Load/Unload Cycles** attribute 193
20 +6. **Verify Error Uncorrected**
21
21 -8. **Temperature** attribute 194
22 +7. **Temperature**
23
23 -9. **Current Pending Sectors** attribute 197
24
25 -10. **Off-Line Uncorrectable** attribute 198
25 +For ATA devices:
26 +1. **Read Error Rate**
27
27 -11. **Write Error Rate** attribute 200
28 +2. **Seek Error Rate**
29 +
30 +3. **Soft Read Error Rate**
31 +
32 +4. **Write Error Rate**
33 +
34 +5. **SATA Interface Downshift**
35 +
36 +6. **UDMA CRC Error Count**
37 +
38 +7. **Throughput Performance**
39 +
40 +8. **Seek Time Performance**
41 +
42 +9. **Start/Stop Count**
43 +
44 +10. **Power-On Hours Count**
45 +
46 +11. **Power Cycle Count**
47 +
48 +12. **Unexpected Power Loss**
49 +
50 +13. **Spin-Up Time**
51 +
52 +14. **Spin-up Retries**
53 +
54 +15. **Calibration Retries**
55 +
56 +16. **Temperature**
57 +
58 +17. **Reallocated Sectors Count**
59 +
60 +18. **Reserved Block Count**
61 +
62 +19. **Program Fail Count**
63 +
64 +20. **Erase Fail Count**
65 +
66 +21. **Wear Leveller Worst Case Erase Count**
67 +
68 +22. **Unused Reserved NAND Blocks**
69 +
70 +23. **Reallocation Event Count**
71 +
72 +24. **Current Pending Sector Count**
73 +
74 +25. **Offline Uncorrectable Sector Count**
75 +
76 +26. **Percent Lifetime Used**
77 +
78 +### prerequisite
79 +`smartd` must be running with `-A` option to write smartd attribute information to files.
80 +
81 +For this you need to set `smartd_opts` (or `SMARTD_ARGS`, check _smartd.service_ content) in `/etc/default/smartmontools`:
82 +
83 +
84 +```
85 +# dump smartd attrs info every 600 seconds
86 +smartd_opts="-A /var/log/smartd/ -i 600"
87 +```
88 +
89 +
90 +`smartd` appends logs at every run. It's strongly recommended to use `logrotate` for smartd files.
91
92 ### configuration
93
@@ -33,6 +96,6 @@ local:
96 log_path : '/var/log/smartd/'
97 ```
98
36 -If no configuration is given, module will attempt to read log files in /var/log/smartd/ directory.
99 +If no configuration is given, module will attempt to read log files in `/var/log/smartd/` directory.
100
101 ---
collectors/python.d.plugin/smartd_log/smartd_log.chart.py
+641 -270
@@ -6,182 +6,537 @@
6 import os
7 import re
8
9 -from collections import namedtuple
9 +from copy import deepcopy
10 from time import time
11
12 from bases.collection import read_last_line
13 from bases.FrameworkServices.SimpleService import SimpleService
14
15 -# charts order (can be overridden if you want less charts, or different order)
16 -ORDER = ['1', '4', '5', '7', '9', '12', '193', '194', '197', '198', '200']
17 -
18 -SMART_ATTR = {
19 - '1': 'Read Error Rate',
20 - '2': 'Throughput Performance',
21 - '3': 'Spin-Up Time',
22 - '4': 'Start/Stop Count',
23 - '5': 'Reallocated Sectors Count',
24 - '6': 'Read Channel Margin',
25 - '7': 'Seek Error Rate',
26 - '8': 'Seek Time Performance',
27 - '9': 'Power-On Hours Count',
28 - '10': 'Spin-up Retries',
29 - '11': 'Calibration Retries',
30 - '12': 'Power Cycle Count',
31 - '13': 'Soft Read Error Rate',
32 - '100': 'Erase/Program Cycles',
33 - '103': 'Translation Table Rebuild',
34 - '108': 'Unknown (108)',
35 - '170': 'Reserved Block Count',
36 - '171': 'Program Fail Count',
37 - '172': 'Erase Fail Count',
38 - '173': 'Wear Leveller Worst Case Erase Count',
39 - '174': 'Unexpected Power Loss',
40 - '175': 'Program Fail Count',
41 - '176': 'Erase Fail Count',
42 - '177': 'Wear Leveling Count',
43 - '178': 'Used Reserved Block Count',
44 - '179': 'Used Reserved Block Count',
45 - '180': 'Unused Reserved Block Count',
46 - '181': 'Program Fail Count',
47 - '182': 'Erase Fail Count',
48 - '183': 'SATA Downshifts',
49 - '184': 'End-to-End error',
50 - '185': 'Head Stability',
51 - '186': 'Induced Op-Vibration Detection',
52 - '187': 'Reported Uncorrectable Errors',
53 - '188': 'Command Timeout',
54 - '189': 'High Fly Writes',
55 - '190': 'Temperature',
56 - '191': 'G-Sense Errors',
57 - '192': 'Power-Off Retract Cycles',
58 - '193': 'Load/Unload Cycles',
59 - '194': 'Temperature',
60 - '195': 'Hardware ECC Recovered',
61 - '196': 'Reallocation Events',
62 - '197': 'Current Pending Sectors',
63 - '198': 'Off-line Uncorrectable',
64 - '199': 'UDMA CRC Error Rate',
65 - '200': 'Write Error Rate',
66 - '201': 'Soft Read Errors',
67 - '202': 'Data Address Mark Errors',
68 - '203': 'Run Out Cancel',
69 - '204': 'Soft ECC Corrections',
70 - '205': 'Thermal Asperity Rate',
71 - '206': 'Flying Height',
72 - '207': 'Spin High Current',
73 - '209': 'Offline Seek Performance',
74 - '220': 'Disk Shift',
75 - '221': 'G-Sense Error Rate',
76 - '222': 'Loaded Hours',
77 - '223': 'Load/Unload Retries',
78 - '224': 'Load Friction',
79 - '225': 'Load/Unload Cycles',
80 - '226': 'Load-in Time',
81 - '227': 'Torque Amplification Count',
82 - '228': 'Power-Off Retracts',
83 - '230': 'GMR Head Amplitude',
84 - '231': 'Temperature',
85 - '232': 'Available Reserved Space',
86 - '233': 'Media Wearout Indicator',
87 - '240': 'Head Flying Hours',
88 - '241': 'Total LBAs Written',
89 - '242': 'Total LBAs Read',
90 - '250': 'Read Error Retry Rate'
91 -}
92 -
93 -LIMIT = namedtuple('LIMIT', ['min', 'max'])
94 -
95 -LIMITS = {
96 - '194': LIMIT(0, 200)
97 -}
15
99 -RESCAN_INTERVAL = 60
100 -
101 -REGEX = re.compile(
16 +INCREMENTAL = 'incremental'
17 +ABSOLUTE = 'absolute'
18 +
19 +ATA = 'ata'
20 +SCSI = 'scsi'
21 +CSV = '.csv'
22 +
23 +DEF_RESCAN_INTERVAL = 60
24 +DEF_AGE = 30
25 +DEF_PATH = '/var/log/smartd'
26 +
27 +ATTR1 = '1'
28 +ATTR2 = '2'
29 +ATTR3 = '3'
30 +ATTR4 = '4'
31 +ATTR5 = '5'
32 +ATTR7 = '7'
33 +ATTR8 = '8'
34 +ATTR9 = '9'
35 +ATTR10 = '10'
36 +ATTR11 = '11'
37 +ATTR12 = '12'
38 +ATTR13 = '13'
39 +ATTR170 = '170'
40 +ATTR171 = '171'
41 +ATTR172 = '172'
42 +ATTR173 = '173'
43 +ATTR174 = '174'
44 +ATTR180 = '180'
45 +ATTR183 = '183'
46 +ATTR190 = '190'
47 +ATTR194 = '194'
48 +ATTR196 = '196'
49 +ATTR197 = '197'
50 +ATTR198 = '198'
51 +ATTR199 = '199'
52 +ATTR202 = '202'
53 +ATTR206 = '206'
54 +ATTR_READ_ERR_COR = 'read-total-err-corrected'
55 +ATTR_READ_ERR_UNC = 'read-total-unc-errors'
56 +ATTR_WRITE_ERR_COR = 'write-total-err-corrected'
57 +ATTR_WRITE_ERR_UNC = 'write-total-unc-errors'
58 +ATTR_VERIFY_ERR_COR = 'verify-total-err-corrected'
59 +ATTR_VERIFY_ERR_UNC = 'verify-total-unc-errors'
60 +ATTR_TEMPERATURE = 'temperature'
61 +
62 +
63 +RE_ATA = re.compile(
64 '(\d+);' # attribute
65 '(\d+);' # normalized value
66 '(\d+)', # raw value
67 re.X
68 )
69
70 +RE_SCSI = re.compile(
71 + '([a-z-]+);' # attribute
72 + '([0-9.]+)', # raw value
73 + re.X
74 +)
75
109 -def chart_template(chart_name):
110 - units, attr_id = chart_name.split('_')[-2:]
111 - title = '{value_type} {description}'.format(value_type=units.capitalize(),
112 - description=SMART_ATTR[attr_id])
113 - family = SMART_ATTR[attr_id].lower()
114 -
115 - return {
116 - chart_name: {
117 - 'options': [None, title, units, family, 'smartd_log.' + chart_name, 'line'],
118 - 'lines': []
119 - }
76 +ORDER = [
77 + # errors
78 + 'read_error_rate',
79 + 'seek_error_rate',
80 + 'soft_read_error_rate',
81 + 'write_error_rate',
82 + 'read_total_err_corrected',
83 + 'read_total_unc_errors',
84 + 'write_total_err_corrected',
85 + 'write_total_unc_errors',
86 + 'verify_total_err_corrected',
87 + 'verify_total_unc_errors',
88 + # external failure
89 + 'sata_interface_downshift',
90 + 'udma_crc_error_count',
91 + # performance
92 + 'throughput_performance',
93 + 'seek_time_performance',
94 + # power
95 + 'start_stop_count',
96 + 'power_on_hours_count',
97 + 'power_cycle_count',
98 + 'unexpected_power_loss',
99 + # spin
100 + 'spin_up_time',
101 + 'spin_up_retries',
102 + 'calibration_retries',
103 + # temperature
104 + 'airflow_temperature_celsius',
105 + 'temperature_celsius',
106 + # wear
107 + 'reallocated_sectors_count',
108 + 'reserved_block_count',
109 + 'program_fail_count',
110 + 'erase_fail_count',
111 + 'wear_leveller_worst_case_erase_count',
112 + 'unused_reserved_nand_blocks',
113 + 'reallocation_event_count',
114 + 'current_pending_sector_count',
115 + 'offline_uncorrectable_sector_count',
116 + 'percent_lifetime_used',
117 +]
118 +
119 +CHARTS = {
120 + 'read_error_rate': {
121 + 'options': [None, 'Read Error Rate', 'value', 'errors', 'smartd_log.read_error_rate', 'line'],
122 + 'lines': [],
123 + 'attrs': [ATTR1],
124 + 'algo': ABSOLUTE,
125 + },
126 + 'seek_error_rate': {
127 + 'options': [None, 'Seek Error Rate', 'value', 'errors', 'smartd_log.seek_error_rate', 'line'],
128 + 'lines': [],
129 + 'attrs': [ATTR7],
130 + 'algo': ABSOLUTE,
131 + },
132 + 'soft_read_error_rate': {
133 + 'options': [None, 'Soft Read Error Rate', 'errors', 'errors', 'smartd_log.soft_read_error_rate', 'line'],
134 + 'lines': [],
135 + 'attrs': [ATTR13],
136 + 'algo': INCREMENTAL,
137 + },
138 + 'write_error_rate': {
139 + 'options': [None, 'Write Error Rate', 'value', 'errors', 'smartd_log.write_error_rate', 'line'],
140 + 'lines': [],
141 + 'attrs': [ATTR206],
142 + 'algo': ABSOLUTE,
143 + },
144 + 'read_total_err_corrected': {
145 + 'options': [None, 'Read Error Corrected', 'errors', 'errors', 'smartd_log.read_total_err_corrected', 'line'],
146 + 'lines': [],
147 + 'attrs': [ATTR_READ_ERR_COR],
148 + 'algo': INCREMENTAL,
149 + },
150 + 'read_total_unc_errors': {
151 + 'options': [None, 'Read Error Uncorrected', 'errors', 'errors', 'smartd_log.read_total_unc_errors', 'line'],
152 + 'lines': [],
153 + 'attrs': [ATTR_READ_ERR_UNC],
154 + 'algo': INCREMENTAL,
155 + },
156 + 'write_total_err_corrected': {
157 + 'options': [None, 'Write Error Corrected', 'errors', 'errors', 'smartd_log.read_total_err_corrected', 'line'],
158 + 'lines': [],
159 + 'attrs': [ATTR_WRITE_ERR_COR],
160 + 'algo': INCREMENTAL,
161 + },
162 + 'write_total_unc_errors': {
163 + 'options': [None, 'Write Error Uncorrected', 'errors', 'errors', 'smartd_log.write_total_unc_errors', 'line'],
164 + 'lines': [],
165 + 'attrs': [ATTR_WRITE_ERR_UNC],
166 + 'algo': INCREMENTAL,
167 + },
168 + 'verify_total_err_corrected': {
169 + 'options': [None, 'Verify Error Corrected', 'errors', 'errors', 'smartd_log.verify_total_err_corrected',
170 + 'line'],
171 + 'lines': [],
172 + 'attrs': [ATTR_VERIFY_ERR_COR],
173 + 'algo': INCREMENTAL,
174 + },
175 + 'verify_total_unc_errors': {
176 + 'options': [None, 'Verify Error Uncorrected', 'errors', 'errors', 'smartd_log.verify_total_unc_errors', 'line'],
177 + 'lines': [],
178 + 'attrs': [ATTR_VERIFY_ERR_UNC],
179 + 'algo': INCREMENTAL,
180 + },
181 + 'sata_interface_downshift': {
182 + 'options': [None, 'SATA Interface Downshift', 'events', 'external failure',
183 + 'smartd_log.sata_interface_downshift', 'line'],
184 + 'lines': [],
185 + 'attrs': [ATTR183],
186 + 'algo': INCREMENTAL,
187 + },
188 + 'udma_crc_error_count': {
189 + 'options': [None, 'UDMA CRC Error Count', 'errors', 'external failure', 'smartd_log.udma_crc_error_count',
190 + 'line'],
191 + 'lines': [],
192 + 'attrs': [ATTR199],
193 + 'algo': INCREMENTAL,
194 + },
195 + 'throughput_performance': {
196 + 'options': [None, 'Throughput Performance', 'value', 'performance', 'smartd_log.throughput_performance',
197 + 'line'],
198 + 'lines': [],
199 + 'attrs': [ATTR2],
200 + 'algo': ABSOLUTE,
201 + },
202 + 'seek_time_performance': {
203 + 'options': [None, 'Seek Time Performance', 'value', 'performance', 'smartd_log.seek_time_performance', 'line'],
204 + 'lines': [],
205 + 'attrs': [ATTR8],
206 + 'algo': ABSOLUTE,
207 + },
208 + 'start_stop_count': {
209 + 'options': [None, 'Start/Stop Count', 'events', 'power', 'smartd_log.start_stop_count', 'line'],
210 + 'lines': [],
211 + 'attrs': [ATTR4],
212 + 'algo': ABSOLUTE,
213 + },
214 + 'power_on_hours_count': {
215 + 'options': [None, 'Power-On Hours Count', 'hours', 'power', 'smartd_log.power_on_hours_count', 'line'],
216 + 'lines': [],
217 + 'attrs': [ATTR9],
218 + 'algo': ABSOLUTE,
219 + },
220 + 'power_cycle_count': {
221 + 'options': [None, 'Power Cycle Count', 'events', 'power', 'smartd_log.power_cycle_count', 'line'],
222 + 'lines': [],
223 + 'attrs': [ATTR12],
224 + 'algo': ABSOLUTE,
225 + },
226 + 'unexpected_power_loss': {
227 + 'options': [None, 'Unexpected Power Loss', 'events', 'power', 'smartd_log.unexpected_power_loss', 'line'],
228 + 'lines': [],
229 + 'attrs': [ATTR174],
230 + 'algo': ABSOLUTE,
231 + },
232 + 'spin_up_time': {
233 + 'options': [None, 'Spin-Up Time', 'ms', 'spin', 'smartd_log.spin_up_time', 'line'],
234 + 'lines': [],
235 + 'attrs': [ATTR3],
236 + 'algo': ABSOLUTE,
237 + },
238 + 'spin_up_retries': {
239 + 'options': [None, 'Spin-up Retries', 'retries', 'spin', 'smartd_log.spin_up_retries', 'line'],
240 + 'lines': [],
241 + 'attrs': [ATTR10],
242 + 'algo': INCREMENTAL,
243 + },
244 + 'calibration_retries': {
245 + 'options': [None, 'Calibration Retries', 'retries', 'spin', 'smartd_log.calibration_retries', 'line'],
246 + 'lines': [],
247 + 'attrs': [ATTR11],
248 + 'algo': INCREMENTAL,
249 + },
250 + 'airflow_temperature_celsius': {
251 + 'options': [None, 'Airflow Temperature Celsius', 'celsius', 'temperature',
252 + 'smartd_log.airflow_temperature_celsius', 'line'],
253 + 'lines': [],
254 + 'attrs': [ATTR190],
255 + 'algo': ABSOLUTE,
256 + },
257 + 'temperature_celsius': {
258 + 'options': [None, 'Temperature', 'celsius', 'temperature', 'smartd_log.temperature_celsius', 'line'],
259 + 'lines': [],
260 + 'attrs': [ATTR194, ATTR_TEMPERATURE],
261 + 'algo': ABSOLUTE,
262 + },
263 + 'reallocated_sectors_count': {
264 + 'options': [None, 'Reallocated Sectors Count', 'sectors', 'wear', 'smartd_log.reallocated_sectors_count',
265 + 'line'],
266 + 'lines': [],
267 + 'attrs': [ATTR5],
268 + 'algo': INCREMENTAL,
269 + },
270 + 'reserved_block_count': {
271 + 'options': [None, 'Reserved Block Count', '%', 'wear', 'smartd_log.reserved_block_count', 'line'],
272 + 'lines': [],
273 + 'attrs': [ATTR170],
274 + 'algo': ABSOLUTE,
275 + },
276 + 'program_fail_count': {
277 + 'options': [None, 'Program Fail Count', 'errors', 'wear', 'smartd_log.program_fail_count', 'line'],
278 + 'lines': [],
279 + 'attrs': [ATTR171],
280 + 'algo': INCREMENTAL,
281 + },
282 + 'erase_fail_count': {
283 + 'options': [None, 'Erase Fail Count', 'failures', 'wear', 'smartd_log.erase_fail_count', 'line'],
284 + 'lines': [],
285 + 'attrs': [ATTR172],
286 + 'algo': INCREMENTAL,
287 + },
288 + 'wear_leveller_worst_case_erase_count': {
289 + 'options': [None, 'Wear Leveller Worst Case Erase Count', 'erases', 'wear',
290 + 'smartd_log.wear_leveller_worst_case_erase_count', 'line'],
291 + 'lines': [],
292 + 'attrs': [ATTR173],
293 + 'algo': ABSOLUTE,
294 + },
295 + 'unused_reserved_nand_blocks': {
296 + 'options': [None, 'Unused Reserved NAND Blocks', 'blocks', 'wear', 'smartd_log.unused_reserved_nand_blocks',
297 + 'line'],
298 + 'lines': [],
299 + 'attrs': [ATTR180],
300 + 'algo': ABSOLUTE,
301 + },
302 + 'reallocation_event_count': {
303 + 'options': [None, 'Reallocation Event Count', 'events', 'wear', 'smartd_log.reallocation_event_count', 'line'],
304 + 'lines': [],
305 + 'attrs': [ATTR196],
306 + 'algo': INCREMENTAL,
307 + },
308 + 'current_pending_sector_count': {
309 + 'options': [None, 'Current Pending Sector Count', 'sectors', 'wear', 'smartd_log.current_pending_sector_count',
310 + 'line'],
311 + 'lines': [],
312 + 'attrs': [ATTR197],
313 + 'algo': ABSOLUTE,
314 + },
315 + 'offline_uncorrectable_sector_count': {
316 + 'options': [None, 'Offline Uncorrectable Sector Count', 'sectors', 'wear',
317 + 'smartd_log.offline_uncorrectable_sector_count', 'line'],
318 + 'lines': [],
319 + 'attrs': [ATTR198],
320 + 'algo': ABSOLUTE,
321 +
322 + },
323 + 'percent_lifetime_used': {
324 + 'options': [None, 'Percent Lifetime Used', '%', 'wear', 'smartd_log.percent_lifetime_used', 'line'],
325 + 'lines': [],
326 + 'attrs': [ATTR202],
327 + 'algo': ABSOLUTE,
328 }
329 +}
330 +
331 +# NOTE: 'parse_temp' decodes ATA 194 raw value. Not heavily tested. Written by @Ferroin
332 +# C code:
333 +# https://github.com/smartmontools/smartmontools/blob/master/smartmontools/atacmds.cpp#L2051
334 +#
335 +# Calling 'parse_temp' on the raw value will return a 4-tuple, containing
336 +# * temperature
337 +# * minimum
338 +# * maximum
339 +# * over-temperature count
340 +# substituting None for values it can't decode.
341 +#
342 +# Example:
343 +# >>> parse_temp(42952491042)
344 +# >>> (34, 10, 43, None)
345 +#
346 +#
347 +# def check_temp_word(i):
348 +# if i <= 0x7F:
349 +# return 0x11
350 +# elif i <= 0xFF:
351 +# return 0x01
352 +# elif 0xFF80 <= i:
353 +# return 0x10
354 +# return 0x00
355 +#
356 +#
357 +# def check_temp_range(t, b0, b1):
358 +# if b0 > b1:
359 +# t0, t1 = b1, b0
360 +# else:
361 +# t0, t1 = b0, b1
362 +#
363 +# if all([
364 +# -60 <= t0,
365 +# t0 <= t,
366 +# t <= t1,
367 +# t1 <= 120,
368 +# not (t0 == -1 and t1 <= 0)
369 +# ]):
370 +# return t0, t1
371 +# return None, None
372 +#
373 +#
374 +# def parse_temp(raw):
375 +# byte = list()
376 +# word = list()
377 +# for i in range(0, 6):
378 +# byte.append(0xFF & (raw >> (i * 8)))
379 +# for i in range(0, 3):
380 +# word.append(0xFFFF & (raw >> (i * 16)))
381 +#
382 +# ctwd = check_temp_word(word[0])
383 +#
384 +# if not word[2]:
385 +# if ctwd and not word[1]:
386 +# # byte[0] is temp, no other data
387 +# return byte[0], None, None, None
388 +#
389 +# if ctwd and all(check_temp_range(byte[0], byte[2], byte[3])):
390 +# # byte[0] is temp, byte[2] is max or min, byte[3] is min or max
391 +# trange = check_temp_range(byte[0], byte[2], byte[3])
392 +# return byte[0], trange[0], trange[1], None
393 +#
394 +# if ctwd and all(check_temp_range(byte[0], byte[1], byte[2])):
395 +# # byte[0] is temp, byte[1] is max or min, byte[2] is min or max
396 +# trange = check_temp_range(byte[0], byte[1], byte[2])
397 +# return byte[0], trange[0], trange[1], None
398 +#
399 +# return None, None, None, None
400 +#
401 +# if ctwd:
402 +# if all(
403 +# [
404 +# ctwd & check_temp_word(word[1]) & check_temp_word(word[2]) != 0x00,
405 +# all(check_temp_range(byte[0], byte[2], byte[4])),
406 +# ]
407 +# ):
408 +# # byte[0] is temp, byte[2] is max or min, byte[4] is min or max
409 +# trange = check_temp_range(byte[0], byte[2], byte[4])
410 +# return byte[0], trange[0], trange[1], None
411 +# else:
412 +# trange = check_temp_range(byte[0], byte[2], byte[3])
413 +# if word[2] < 0x7FFF and all(trange) and trange[1] >= 40:
414 +# # byte[0] is temp, byte[2] is max or min, byte[3] is min or max, word[2] is overtemp count
415 +# return byte[0], trange[0], trange[1], word[2]
416 +# # no data
417 +# return None, None, None, None
418 +
419 +
420 +CHARTED_ATTRS = dict((attr, k) for k, v in CHARTS.items() for attr in v['attrs'])
421 +
422 +
423 +class BaseAtaSmartAttribute:
424 + def __init__(self, name, normalized_value, raw_value):
425 + self.name = name
426 + self.normalized_value = normalized_value
427 + self.raw_value = raw_value
428 +
429 + def value(self):
430 + raise NotImplementedError
431 +
432 +
433 +class AtaRaw(BaseAtaSmartAttribute):
434 + def value(self):
435 + return self.raw_value
436 +
437 +
438 +class AtaNormalized(BaseAtaSmartAttribute):
439 + def value(self):
440 + return self.normalized_value
441 +
442 +
443 +class Ata9(BaseAtaSmartAttribute):
444 + def value(self):
445 + value = int(self.raw_value)
446 + if value > 1e6:
447 + return value & 0xFFFF
448 + return value
449 +
450 +
451 +class Ata190(BaseAtaSmartAttribute):
452 + def value(self):
453 + return 100 - int(self.normalized_value)
454 +
455 +
456 +class BaseSCSISmartAttribute:
457 + def __init__(self, name, raw_value):
458 + self.name = name
459 + self.raw_value = raw_value
460 +
461 + def value(self):
462 + raise NotImplementedError
463 +
464 +
465 +class SCSIRaw(BaseSCSISmartAttribute):
466 + def value(self):
467 + return self.raw_value
468 +
469 +
470 +def ata_attribute_factory(value):
471 + name = value[0]
472 +
473 + if name == ATTR9:
474 + return Ata9(*value)
475 + elif name == ATTR190:
476 + return Ata190(*value)
477 + elif name in [
478 + ATTR1,
479 + ATTR7,
480 + ATTR194,
481 + ATTR202,
482 + ATTR206,
483 + ]:
484 + return AtaNormalized(*value)
485 +
486 + return AtaRaw(*value)
487
488
123 -def handle_os_error(method):
124 - def on_call(*args):
125 - try:
126 - return method(*args)
127 - except OSError:
128 - return None
129 - return on_call
489 +def scsi_attribute_factory(value):
490 + return SCSIRaw(*value)
491
492
132 -class SmartAttribute(object):
133 - def __init__(self, idx, normalized, raw):
134 - self.id = idx
135 - self.normalized = normalized
136 - self._raw = raw
493 +def attribute_factory(value):
494 + name = value[0]
495 + if name.isdigit():
496 + return ata_attribute_factory(value)
497 + return scsi_attribute_factory(value)
498
138 - @property
139 - def raw(self):
140 - if self.id in LIMITS:
141 - limit = LIMITS[self.id]
142 - if limit.min <= int(self._raw) <= limit.max:
143 - return self._raw
144 - return None
145 - return self._raw
499
147 - @raw.setter
148 - def raw(self, value):
149 - self._raw = value
500 +def handle_error(*errors):
501 + def on_method(method):
502 + def on_call(*args):
503 + try:
504 + return method(*args)
505 + except errors:
506 + return None
507 + return on_call
508 + return on_method
509
510
511 class DiskLogFile:
153 - def __init__(self, path):
154 - self.path = path
155 - self.size = os.path.getsize(path)
512 + def __init__(self, full_path):
513 + self.path = full_path
514 + self.size = os.path.getsize(full_path)
515
157 - @handle_os_error
516 + @handle_error(OSError)
517 def is_changed(self):
159 - new_size = os.path.getsize(self.path)
160 - old_size, self.size = self.size, new_size
161 -
162 - return new_size != old_size and new_size
518 + return self.size != os.path.getsize(self.path)
519
164 - @staticmethod
165 - @handle_os_error
166 - def is_valid(log_file, exclude):
167 - return all([log_file.endswith('.csv'),
168 - not [p for p in exclude if p in log_file],
169 - os.access(log_file, os.R_OK),
170 - os.path.getsize(log_file)])
520 + @handle_error(OSError)
521 + def is_active(self, current_time, limit):
522 + return (current_time - os.path.getmtime(self.path)) / 60 < limit
523
524 + @handle_error(OSError)
525 + def read(self):
526 + self.size = os.path.getsize(self.path)
527 + return read_last_line(self.path)
528
173 -class Disk:
174 - def __init__(self, full_path, age):
175 - self.log_file = DiskLogFile(full_path)
176 - self.name = os.path.basename(full_path).split('.')[-3]
177 - self.age = int(age)
178 - self.status = True
179 - self.attributes = dict()
529
181 - self.get_attributes()
530 +class BaseDisk:
531 + def __init__(self, name, log_file):
532 + self.name = re.sub(r'_+', '_', name)
533 + self.log_file = log_file
534 + self.attrs = list()
535 + self.alive = True
536 + self.charted = False
537
538 def __eq__(self, other):
184 - if isinstance(other, Disk):
539 + if isinstance(other, BaseDisk):
540 return self.name == other.name
541 return self.name == other
542
@@ -191,163 +546,179 @@ class Disk:
546 def __hash__(self):
547 return hash(repr(self))
548
194 - @handle_os_error
195 - def is_active(self):
196 - return (time() - os.path.getmtime(self.log_file.path)) / 60 < self.age
549 + def parser(self, data):
550 + raise NotImplementedError
551
198 - @handle_os_error
199 - def get_attributes(self):
200 - last_line = read_last_line(self.log_file.path)
201 - self.attributes = dict((attr, SmartAttribute(attr, normalized, raw)) for attr, normalized, raw
202 - in REGEX.findall(last_line))
203 - return True
552 + @handle_error(TypeError)
553 + def populate_attrs(self):
554 + self.attrs.clear()
555 + line = self.log_file.read()
556 + for value in self.parser(line):
557 + self.attrs.append(attribute_factory(value))
558 +
559 + return len(self.attrs)
560
561 def data(self):
562 data = dict()
207 - for attr in self.attributes.values():
208 - data['_'.join([self.name, 'normalized', attr.id])] = attr.normalized
209 - if attr.raw is not None:
210 - data['_'.join([self.name, 'raw', attr.id])] = attr.raw
563 + for attr in self.attrs:
564 + data['{0}_{1}'.format(self.name, attr.name)] = attr.value()
565 return data
566
567
214 -class Service(SimpleService):
215 - def __init__(self, configuration=None, name=None):
216 - SimpleService.__init__(self, configuration=configuration, name=name)
217 - self.log_path = self.configuration.get('log_path', '/var/log/smartd')
218 - self.raw = self.configuration.get('raw_values', True)
219 - self.exclude = self.configuration.get('exclude_disks', str()).split()
220 - self.age = self.configuration.get('age', 30)
568 +class ATADisk(BaseDisk):
569 + def parser(self, data):
570 + return RE_ATA.findall(data)
571
222 - self.runs = 0
223 - self.disks = list()
224 - self.order = list()
225 - self.definitions = dict()
572
227 - def check(self):
228 - self.disks = self.scan()
573 +class SCSIDisk(BaseDisk):
574 + def parser(self, data):
575 + return RE_SCSI.findall(data)
576
230 - if not self.disks:
231 - return None
577
233 - user_defined_sa = self.configuration.get('smart_attributes')
578 +class Service(SimpleService):
579 + def __init__(self, configuration=None, name=None):
580 + SimpleService.__init__(self, configuration=configuration, name=name)
581 + self.order = ORDER
582 + self.definitions = deepcopy(CHARTS)
583
235 - if user_defined_sa:
236 - order = user_defined_sa.split() or ORDER
237 - else:
238 - order = ORDER
584 + self.log_path = configuration.get('log_path', DEF_PATH)
585 + self.age = configuration.get('age', DEF_AGE)
586 + self.exclude = configuration.get('exclude_disks', str()).split()
587
240 - self.create_charts(order)
588 + self.disks = list()
589 + self.runs = 0
590
242 - return True
591 + def check(self):
592 + return self.scan() > 0
593
594 def get_data(self):
595 self.runs += 1
596
247 - if self.runs % RESCAN_INTERVAL == 0:
248 - self.cleanup_and_rescan()
597 + if self.runs % DEF_RESCAN_INTERVAL == 0:
598 + self.cleanup()
599 + self.scan()
600
601 data = dict()
602
603 for disk in self.disks:
253 -
254 - if not disk.status:
604 + if not disk.alive:
605 continue
606
607 + if not disk.charted:
608 + self.add_disk_to_charts(disk)
609 +
610 changed = disk.log_file.is_changed()
611
259 - # True = changed, False = unchanged, None = Exception
612 if changed is None:
261 - disk.status = False
613 + disk.alive = False
614 continue
615
264 - if changed:
265 - success = disk.get_attributes()
266 - if not success:
267 - disk.status = False
268 - continue
616 + if changed and disk.populate_attrs() is None:
617 + disk.alive = False
618 + continue
619
620 data.update(disk.data())
621
272 - return data or None
273 -
274 - def create_charts(self, order):
275 - for attr in order:
276 - raw_name, normalized_name = 'attr_id_raw_' + attr, 'attr_id_normalized_' + attr
277 - raw, normalized = chart_template(raw_name), chart_template(normalized_name)
278 - self.order.extend([normalized_name, raw_name])
279 - self.definitions.update(raw)
280 - self.definitions.update(normalized)
281 -
282 - for disk in self.disks:
283 - if attr not in disk.attributes:
284 - self.debug("'{disk}' has no attribute '{attr_id}'".format(disk=disk.name,
285 - attr_id=attr))
286 - continue
287 - normalized[normalized_name]['lines'].append(['_'.join([disk.name, 'normalized', attr]), disk.name])
288 -
289 - if not self.raw:
290 - continue
291 -
292 - if disk.attributes[attr].raw is not None:
293 - raw[raw_name]['lines'].append(['_'.join([disk.name, 'raw', attr]), disk.name])
294 - continue
295 - self.debug("'{disk}' attribute '{attr_id}' value not in {limits}".format(disk=disk.name,
296 - attr_id=attr,
297 - limits=LIMITS[attr]))
298 -
299 - def cleanup_and_rescan(self):
300 - self.cleanup()
301 - new_disks = self.scan(only_new=True)
302 -
303 - for disk in new_disks:
304 - valid = False
305 -
306 - for chart in self.charts:
307 - value_type, idx = chart.id.split('_')[2:]
308 -
309 - if idx in disk.attributes:
310 - valid = True
311 - dimension_id = '_'.join([disk.name, value_type, idx])
312 -
313 - if dimension_id in chart:
314 - chart.hide_dimension(dimension_id=dimension_id, reverse=True)
315 - else:
316 - chart.add_dimension([dimension_id, disk.name])
317 - if valid:
318 - self.disks.append(disk)
622 + return data
623
624 def cleanup(self):
321 - for disk in self.disks:
625 + current_time = time()
626 + for disk in self.disks[:]:
627 + if any(
628 + [
629 + not disk.alive,
630 + not disk.log_file.is_active(current_time, self.age),
631 + ]
632 + ):
633 + self.disks.remove(disk.name)
634 + self.remove_disk_from_charts(disk)
635 +
636 + def scan(self):
637 + self.debug('scanning {0}'.format(self.log_path))
638 + current_time = time()
639 +
640 + for full_name in os.listdir(self.log_path):
641 + disk = self.create_disk_from_file(full_name, current_time)
642 + if not disk:
643 + continue
644 + self.disks.append(disk)
645 +
646 + return len(self.disks)
647 +
648 + def create_disk_from_file(self, full_name, current_time):
649 + name = os.path.basename(full_name).split('.')[-3]
650 + path = os.path.join(self.log_path, full_name)
651 +
652 + if name in self.disks:
653 + return None
654 +
655 + if [p for p in self.exclude if p in name]:
656 + return None
657 +
658 + if not full_name.endswith(CSV):
659 + self.debug('skipping {0}: not a csv file'.format(full_name))
660 + return None
661 +
662 + if not os.access(path, os.R_OK):
663 + self.debug('skipping {0}: not readable'.format(full_name))
664 + return None
665 +
666 + if os.path.getsize(path) == 0:
667 + self.debug('skipping {0}: zero size'.format(full_name))
668 + return None
669 +
670 + if (current_time - os.path.getmtime(path)) / 60 > self.age:
671 + self.debug('skipping {0}: haven\'t been updated for last {1} minutes'.format(full_name, self.age))
672 + return None
673 +
674 + if ATA in full_name:
675 + disk = ATADisk(name, DiskLogFile(path))
676 + elif SCSI in full_name:
677 + disk = SCSIDisk(name, DiskLogFile(path))
678 + else:
679 + self.debug('skipping {0}: unknown type'.format(full_name))
680 + return None
681 +
682 + disk.populate_attrs()
683 + if not disk.attrs:
684 + self.error('skipping {0}: parsing failed'.format(full_name))
685 + return None
686 +
687 + self.debug('added {0}'.format(full_name))
688 + return disk
689 +
690 + def add_disk_to_charts(self, disk):
691 + if len(self.charts) == 0 or disk.charted:
692 + return
693 + disk.charted = True
694 +
695 + for attr in disk.attrs:
696 + chart_id = CHARTED_ATTRS.get(attr.name)
697 +
698 + if not chart_id or chart_id not in self.charts:
699 + continue
700 +
701 + chart = self.charts[chart_id]
702 + dim = [
703 + '{0}_{1}'.format(disk.name, attr.name),
704 + disk.name,
705 + CHARTS[chart_id]['algo'],
706 + ]
707 +
708 + if dim[0] in self.charts[chart_id].dimensions:
709 + chart.hide_dimension(dim[0], reverse=True)
710 + else:
711 + chart.add_dimension(dim)
712 +
713 + def remove_disk_from_charts(self, disk):
714 + if len(self.charts) == 0 or not disk.charted:
715 + return
716 +
717 + for attr in disk.attrs:
718 + chart_id = CHARTED_ATTRS.get(attr.name)
719 +
720 + if not chart_id or chart_id not in self.charts:
721 + continue
722
323 - if not disk.is_active():
324 - disk.status = False
325 - if not disk.status:
326 - for chart in self.charts:
327 - dimension_id = '_'.join([disk.name, chart.id[8:]])
328 - chart.hide_dimension(dimension_id=dimension_id)
329 -
330 - self.disks = [disk for disk in self.disks if disk.status]
331 -
332 - def scan(self, only_new=None):
333 - new_disks = list()
334 - for f in os.listdir(self.log_path):
335 - full_path = os.path.join(self.log_path, f)
336 -
337 - if DiskLogFile.is_valid(full_path, self.exclude):
338 - disk = Disk(full_path, self.age)
339 -
340 - active = disk.is_active()
341 - if active is None:
342 - continue
343 - if active:
344 - if not only_new:
345 - new_disks.append(disk)
346 - else:
347 - if disk not in self.disks:
348 - new_disks.append(disk)
349 - else:
350 - if not only_new:
351 - self.debug("'{disk}' not updated in the last {age} minutes, "
352 - "skipping it.".format(disk=disk.name, age=self.age))
353 - return new_disks
723 + # TODO: can't delete dimension
724 + self.charts[chart_id].hide_dimension('{0}_{1}'.format(disk.name, attr.name))
collectors/python.d.plugin/smartd_log/smartd_log.conf
+2 -23
@@ -63,28 +63,7 @@
63 #
64 # Additionally to the above, smartd_log also supports the following:
65 #
66 -# log_path: '/path/to/smartdlogs' # path to smartd log files. Default is /var/log/smartd
67 -# raw_values: yes # enable/disable raw values charts. Enabled by default.
68 -# smart_attributes: '1 2 3 4 44' # smart attributes charts. Default are ['1', '4', '5', '7', '9', '12', '193', '194', '197', '198', '200'].
69 -# exclude_disks: 'PATTERN1 PATTERN2' # space separated patterns. If the pattern is in the drive name, the module will not collect data for it.
70 -#
71 -# ----------------------------------------------------------------------
72 -# Additional information
73 -# Plugin reads smartd log files (-A option).
74 -# You need to add (man smartd) to /etc/default/smartmontools '-i 600 -A /var/log/smartd/' to pass additional options to smartd on startup
75 -# Then restart smartd service and check /path/log/smartdlogs
76 -# ls /var/log/smartd/
77 -# CDC_WD10EZEX_00BN5A0-WD_WCC3F7FLVZS9.ata.csv WDC_WD10EZEX_00BN5A0-WD_WCC3F7FLVZS9.ata.csv ZDC_WD10EZEX_00BN5A0-WD_WCC3F7FLVZS9.ata.csv
78 -#
79 -# Smartd APPEND logs at every run. Its NOT RECOMMENDED to set '-i' option below 60 sec.
80 -# STRONGLY RECOMMENDED to create smartd conf file for logrotate
81 -#
82 -# RAW vs NORMALIZED values
83 -# "Normalized value", commonly referred to as just "value". This is a most universal measurement, on the scale from 0 (bad) to some maximum (good) value.
84 -# Maximum values are typically 100, 200 or 253. Rule of thumb is: high values are good, low values are bad.
85 -#
86 -# "Raw value" - the value of the attribute as it is tracked by the device, before any normalization takes place.
87 -# Some raw numbers provide valuable insight when properly interpreted. These cases will be discussed later on.
88 -# Raw values are typically listed in hexadecimal numbers. The raw value has different structure for different vendors and is often not meaningful as a decimal number.
66 +# log_path: '/path/to/smartd_logs' # path to smartd log files. Default is /var/log/smartd
67 +# exclude_disks: 'PATTERN1 PATTERN2' # space separated patterns. If the pattern is in the drive name, the module will not collect data for it.
68 #
69 # ----------------------------------------------------------------------