remove python.d/varnish (#18499)
Ilya Mashchenko committed
Sep 10, 2024 at 12:43 UTC
b97ccb7b9f0073d9a0abeea4453019208b91164e
6 files changed
-954
CMakeLists.txt
-2
@@ -2900,7 +2900,6 @@ install(FILES
2900
src/collectors/python.d.plugin/pandas/pandas.conf
2901
src/collectors/python.d.plugin/spigotmc/spigotmc.conf
2902
src/collectors/python.d.plugin/traefik/traefik.conf
2903
- src/collectors/python.d.plugin/varnish/varnish.conf
2903
src/collectors/python.d.plugin/zscores/zscores.conf
2904
COMPONENT plugin-pythond
2905
DESTINATION usr/lib/netdata/conf.d/python.d)
@@ -2915,7 +2914,6 @@ install(FILES
2914
src/collectors/python.d.plugin/pandas/pandas.chart.py
2915
src/collectors/python.d.plugin/spigotmc/spigotmc.chart.py
2916
src/collectors/python.d.plugin/traefik/traefik.chart.py
2918
- src/collectors/python.d.plugin/varnish/varnish.chart.py
2917
src/collectors/python.d.plugin/zscores/zscores.chart.py
2918
COMPONENT plugin-pythond
2919
DESTINATION usr/libexec/netdata/python.d)
src/collectors/python.d.plugin/varnish/README.md
deleted
-1
@@ -1 +0,0 @@
1
-integrations/varnish.md
\ No newline at end of file
src/collectors/python.d.plugin/varnish/integrations/varnish.md
deleted
-247
@@ -1,247 +0,0 @@
1
-<!--startmeta
2
-custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/varnish/README.md"
3
-meta_yaml: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/varnish/metadata.yaml"
4
-sidebar_label: "Varnish"
5
-learn_status: "Published"
6
-learn_rel_path: "Collecting Metrics/Web Servers and Web Proxies"
7
-most_popular: False
8
-message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE"
9
-endmeta-->
10
-
11
-# Varnish
12
-
13
-
14
-<img src="https://netdata.cloud/img/varnish.svg" width="150"/>
15
-
16
-
17
-Plugin: python.d.plugin
18
-Module: varnish
19
-
20
-<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
21
-
22
-## Overview
23
-
24
-This collector monitors Varnish metrics about HTTP accelerator global, Backends (VBE) and Storages (SMF, SMA, MSE) statistics.
25
-
26
-Note that both, Varnish-Cache (free and open source) and Varnish-Plus (Commercial/Enterprise version), are supported.
27
-
28
-
29
-It uses the `varnishstat` tool in order to collect the metrics.
30
-
31
-
32
-This collector is supported on all platforms.
33
-
34
-This collector only supports collecting metrics from a single instance of this integration.
35
-
36
-`netdata` user must be a member of the `varnish` group.
37
-
38
-
39
-### Default Behavior
40
-
41
-#### Auto-Detection
42
-
43
-By default, if the permissions are satisfied, the `varnishstat` tool will be executed on the host.
44
-
45
-#### Limits
46
-
47
-The default configuration for this integration does not impose any limits on data collection.
48
-
49
-#### Performance Impact
50
-
51
-The default configuration for this integration is not expected to impose a significant performance impact on the system.
52
-
53
-
54
-## Metrics
55
-
56
-Metrics grouped by *scope*.
57
-
58
-The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.
59
-
60
-
61
-
62
-### Per Varnish instance
63
-
64
-These metrics refer to the entire monitored application.
65
-
66
-This scope has no labels.
67
-
68
-Metrics:
69
-
70
-| Metric | Dimensions | Unit |
71
-|:------|:----------|:----|
72
-| varnish.session_connection | accepted, dropped | connections/s |
73
-| varnish.client_requests | received | requests/s |
74
-| varnish.all_time_hit_rate | hit, miss, hitpass | percentage |
75
-| varnish.current_poll_hit_rate | hit, miss, hitpass | percentage |
76
-| varnish.cached_objects_expired | objects | expired/s |
77
-| varnish.cached_objects_nuked | objects | nuked/s |
78
-| varnish.threads_total | None | number |
79
-| varnish.threads_statistics | created, failed, limited | threads/s |
80
-| varnish.threads_queue_len | in queue | requests |
81
-| varnish.backend_connections | successful, unhealthy, reused, closed, recycled, failed | connections/s |
82
-| varnish.backend_requests | sent | requests/s |
83
-| varnish.esi_statistics | errors, warnings | problems/s |
84
-| varnish.memory_usage | free, allocated | MiB |
85
-| varnish.uptime | uptime | seconds |
86
-
87
-### Per Backend
88
-
89
-
90
-
91
-This scope has no labels.
92
-
93
-Metrics:
94
-
95
-| Metric | Dimensions | Unit |
96
-|:------|:----------|:----|
97
-| varnish.backend | header, body | kilobits/s |
98
-
99
-### Per Storage
100
-
101
-
102
-
103
-This scope has no labels.
104
-
105
-Metrics:
106
-
107
-| Metric | Dimensions | Unit |
108
-|:------|:----------|:----|
109
-| varnish.storage_usage | free, allocated | KiB |
110
-| varnish.storage_alloc_objs | allocated | objects |
111
-
112
-
113
-
114
-## Alerts
115
-
116
-There are no alerts configured by default for this integration.
117
-
118
-
119
-## Setup
120
-
121
-### Prerequisites
122
-
123
-#### Provide the necessary permissions
124
-
125
-In order for the collector to work, you need to add the `netdata` user to the `varnish` user group, so that it can execute the `varnishstat` tool:
126
-
127
-```
128
-usermod -aG varnish netdata
129
-```
130
-
131
-
132
-
133
-### Configuration
134
-
135
-#### File
136
-
137
-The configuration file name for this integration is `python.d/varnish.conf`.
138
-
139
-
140
-You can edit the configuration file using the `edit-config` script from the
141
-Netdata [config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory).
142
-
143
-```bash
144
-cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
145
-sudo ./edit-config python.d/varnish.conf
146
-```
147
-#### Options
148
-
149
-There are 2 sections:
150
-
151
-* Global variables
152
-* One or more JOBS that can define multiple different instances to monitor.
153
-
154
-The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values.
155
-
156
-Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition.
157
-
158
-Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified.
159
-
160
-
161
-<details open><summary>Config options</summary>
162
-
163
-| Name | Description | Default | Required |
164
-|:----|:-----------|:-------|:--------:|
165
-| instance_name | the name of the varnishd instance to get logs from. If not specified, the local host name is used. | | yes |
166
-| update_every | Sets the default data collection frequency. | 10 | no |
167
-| priority | Controls the order of charts at the netdata dashboard. | 60000 | no |
168
-| autodetection_retry | Sets the job re-check interval in seconds. | 0 | no |
169
-| penalty | Indicates whether to apply penalty to update_every in case of failures. | yes | no |
170
-| name | Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed running at any time. This allows autodetection to try several alternatives and pick the one that works. | | no |
171
-
172
-</details>
173
-
174
-#### Examples
175
-
176
-##### Basic
177
-
178
-An example configuration.
179
-
180
-```yaml
181
-job_name:
182
- instance_name: '<name-of-varnishd-instance>'
183
-
184
-```
185
-
186
-
187
-## Troubleshooting
188
-
189
-### Debug Mode
190
-
191
-
192
-To troubleshoot issues with the `varnish` collector, run the `python.d.plugin` with the debug option enabled. The output
193
-should give you clues as to why the collector isn't working.
194
-
195
-- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on
196
- your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`.
197
-
198
- ```bash
199
- cd /usr/libexec/netdata/plugins.d/
200
- ```
201
-
202
-- Switch to the `netdata` user.
203
-
204
- ```bash
205
- sudo -u netdata -s
206
- ```
207
-
208
-- Run the `python.d.plugin` to debug the collector:
209
-
210
- ```bash
211
- ./python.d.plugin varnish debug trace
212
- ```
213
-
214
-### Getting Logs
215
-
216
-If you're encountering problems with the `varnish` collector, follow these steps to retrieve logs and identify potential issues:
217
-
218
-- **Run the command** specific to your system (systemd, non-systemd, or Docker container).
219
-- **Examine the output** for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem.
220
-
221
-#### System with systemd
222
-
223
-Use the following command to view logs generated since the last Netdata service restart:
224
-
225
-```bash
226
-journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep varnish
227
-```
228
-
229
-#### System without systemd
230
-
231
-Locate the collector log file, typically at `/var/log/netdata/collector.log`, and use `grep` to filter for collector's name:
232
-
233
-```bash
234
-grep varnish /var/log/netdata/collector.log
235
-```
236
-
237
-**Note**: This method shows logs from all restarts. Focus on the **latest entries** for troubleshooting current issues.
238
-
239
-#### Docker Container
240
-
241
-If your Netdata runs in a Docker container named "netdata" (replace if different), use this command:
242
-
243
-```bash
244
-docker logs netdata 2>&1 | grep varnish
245
-```
246
-
247
-
src/collectors/python.d.plugin/varnish/metadata.yaml
deleted
-253
@@ -1,253 +0,0 @@
1
-plugin_name: python.d.plugin
2
-modules:
3
- - meta:
4
- plugin_name: python.d.plugin
5
- module_name: varnish
6
- monitored_instance:
7
- name: Varnish
8
- link: https://varnish-cache.org/
9
- categories:
10
- - data-collection.web-servers-and-web-proxies
11
- icon_filename: 'varnish.svg'
12
- related_resources:
13
- integrations:
14
- list: []
15
- info_provided_to_referring_integrations:
16
- description: ''
17
- keywords:
18
- - varnish
19
- - varnishstat
20
- - varnishd
21
- - cache
22
- - web server
23
- - web cache
24
- most_popular: false
25
- overview:
26
- data_collection:
27
- metrics_description: |
28
- This collector monitors Varnish metrics about HTTP accelerator global, Backends (VBE) and Storages (SMF, SMA, MSE) statistics.
29
-
30
- Note that both, Varnish-Cache (free and open source) and Varnish-Plus (Commercial/Enterprise version), are supported.
31
- method_description: |
32
- It uses the `varnishstat` tool in order to collect the metrics.
33
- supported_platforms:
34
- include: []
35
- exclude: []
36
- multi_instance: false
37
- additional_permissions:
38
- description: |
39
- `netdata` user must be a member of the `varnish` group.
40
- default_behavior:
41
- auto_detection:
42
- description: By default, if the permissions are satisfied, the `varnishstat` tool will be executed on the host.
43
- limits:
44
- description: ''
45
- performance_impact:
46
- description: ''
47
- setup:
48
- prerequisites:
49
- list:
50
- - title: Provide the necessary permissions
51
- description: |
52
- In order for the collector to work, you need to add the `netdata` user to the `varnish` user group, so that it can execute the `varnishstat` tool:
53
-
54
- ```
55
- usermod -aG varnish netdata
56
- ```
57
- configuration:
58
- file:
59
- name: python.d/varnish.conf
60
- description: ''
61
- options:
62
- description: |
63
- There are 2 sections:
64
-
65
- * Global variables
66
- * One or more JOBS that can define multiple different instances to monitor.
67
-
68
- The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values.
69
-
70
- Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition.
71
-
72
- Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified.
73
- folding:
74
- title: Config options
75
- enabled: true
76
- list:
77
- - name: instance_name
78
- description: the name of the varnishd instance to get logs from. If not specified, the local host name is used.
79
- default_value: ""
80
- required: true
81
- - name: update_every
82
- description: Sets the default data collection frequency.
83
- default_value: 10
84
- required: false
85
- - name: priority
86
- description: Controls the order of charts at the netdata dashboard.
87
- default_value: 60000
88
- required: false
89
- - name: autodetection_retry
90
- description: Sets the job re-check interval in seconds.
91
- default_value: 0
92
- required: false
93
- - name: penalty
94
- description: Indicates whether to apply penalty to update_every in case of failures.
95
- default_value: yes
96
- required: false
97
- - name: name
98
- description: Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed running at any time. This allows autodetection to try several alternatives and pick the one that works.
99
- default_value: ''
100
- required: false
101
- examples:
102
- folding:
103
- enabled: true
104
- title: 'Config'
105
- list:
106
- - name: Basic
107
- description: An example configuration.
108
- folding:
109
- enabled: false
110
- config: |
111
- job_name:
112
- instance_name: '<name-of-varnishd-instance>'
113
- troubleshooting:
114
- problems:
115
- list: []
116
- alerts: []
117
- metrics:
118
- folding:
119
- title: Metrics
120
- enabled: false
121
- description: ""
122
- availability: []
123
- scopes:
124
- - name: global
125
- description: "These metrics refer to the entire monitored application."
126
- labels: []
127
- metrics:
128
- - name: varnish.session_connection
129
- description: Connections Statistics
130
- unit: "connections/s"
131
- chart_type: line
132
- dimensions:
133
- - name: accepted
134
- - name: dropped
135
- - name: varnish.client_requests
136
- description: Client Requests
137
- unit: "requests/s"
138
- chart_type: line
139
- dimensions:
140
- - name: received
141
- - name: varnish.all_time_hit_rate
142
- description: All History Hit Rate Ratio
143
- unit: "percentage"
144
- chart_type: stacked
145
- dimensions:
146
- - name: hit
147
- - name: miss
148
- - name: hitpass
149
- - name: varnish.current_poll_hit_rate
150
- description: Current Poll Hit Rate Ratio
151
- unit: "percentage"
152
- chart_type: stacked
153
- dimensions:
154
- - name: hit
155
- - name: miss
156
- - name: hitpass
157
- - name: varnish.cached_objects_expired
158
- description: Expired Objects
159
- unit: "expired/s"
160
- chart_type: line
161
- dimensions:
162
- - name: objects
163
- - name: varnish.cached_objects_nuked
164
- description: Least Recently Used Nuked Objects
165
- unit: "nuked/s"
166
- chart_type: line
167
- dimensions:
168
- - name: objects
169
- - name: varnish.threads_total
170
- description: Number Of Threads In All Pools
171
- unit: "number"
172
- chart_type: line
173
- dimensions:
174
- - name: None
175
- - name: varnish.threads_statistics
176
- description: Threads Statistics
177
- unit: "threads/s"
178
- chart_type: line
179
- dimensions:
180
- - name: created
181
- - name: failed
182
- - name: limited
183
- - name: varnish.threads_queue_len
184
- description: Current Queue Length
185
- unit: "requests"
186
- chart_type: line
187
- dimensions:
188
- - name: in queue
189
- - name: varnish.backend_connections
190
- description: Backend Connections Statistics
191
- unit: "connections/s"
192
- chart_type: line
193
- dimensions:
194
- - name: successful
195
- - name: unhealthy
196
- - name: reused
197
- - name: closed
198
- - name: recycled
199
- - name: failed
200
- - name: varnish.backend_requests
201
- description: Requests To The Backend
202
- unit: "requests/s"
203
- chart_type: line
204
- dimensions:
205
- - name: sent
206
- - name: varnish.esi_statistics
207
- description: ESI Statistics
208
- unit: "problems/s"
209
- chart_type: line
210
- dimensions:
211
- - name: errors
212
- - name: warnings
213
- - name: varnish.memory_usage
214
- description: Memory Usage
215
- unit: "MiB"
216
- chart_type: stacked
217
- dimensions:
218
- - name: free
219
- - name: allocated
220
- - name: varnish.uptime
221
- description: Uptime
222
- unit: "seconds"
223
- chart_type: line
224
- dimensions:
225
- - name: uptime
226
- - name: Backend
227
- description: ""
228
- labels: []
229
- metrics:
230
- - name: varnish.backend
231
- description: Backend {backend_name}
232
- unit: "kilobits/s"
233
- chart_type: area
234
- dimensions:
235
- - name: header
236
- - name: body
237
- - name: Storage
238
- description: ""
239
- labels: []
240
- metrics:
241
- - name: varnish.storage_usage
242
- description: Storage {storage_name} Usage
243
- unit: "KiB"
244
- chart_type: stacked
245
- dimensions:
246
- - name: free
247
- - name: allocated
248
- - name: varnish.storage_alloc_objs
249
- description: Storage {storage_name} Allocated Objects
250
- unit: "objects"
251
- chart_type: line
252
- dimensions:
253
- - name: allocated
src/collectors/python.d.plugin/varnish/varnish.chart.py
deleted
-385
@@ -1,385 +0,0 @@
1
-# -*- coding: utf-8 -*-
2
-# Description: varnish netdata python.d module
3
-# Author: ilyam8
4
-# SPDX-License-Identifier: GPL-3.0-or-later
5
-
6
-import re
7
-
8
-from bases.FrameworkServices.ExecutableService import ExecutableService
9
-from bases.collection import find_binary
10
-
11
-ORDER = [
12
- 'session_connections',
13
- 'client_requests',
14
- 'all_time_hit_rate',
15
- 'current_poll_hit_rate',
16
- 'cached_objects_expired',
17
- 'cached_objects_nuked',
18
- 'threads_total',
19
- 'threads_statistics',
20
- 'threads_queue_len',
21
- 'backend_connections',
22
- 'backend_requests',
23
- 'esi_statistics',
24
- 'memory_usage',
25
- 'uptime'
26
-]
27
-
28
-CHARTS = {
29
- 'session_connections': {
30
- 'options': [None, 'Connections Statistics', 'connections/s',
31
- 'client metrics', 'varnish.session_connection', 'line'],
32
- 'lines': [
33
- ['sess_conn', 'accepted', 'incremental'],
34
- ['sess_dropped', 'dropped', 'incremental']
35
- ]
36
- },
37
- 'client_requests': {
38
- 'options': [None, 'Client Requests', 'requests/s',
39
- 'client metrics', 'varnish.client_requests', 'line'],
40
- 'lines': [
41
- ['client_req', 'received', 'incremental']
42
- ]
43
- },
44
- 'all_time_hit_rate': {
45
- 'options': [None, 'All History Hit Rate Ratio', 'percentage', 'cache performance',
46
- 'varnish.all_time_hit_rate', 'stacked'],
47
- 'lines': [
48
- ['cache_hit', 'hit', 'percentage-of-absolute-row'],
49
- ['cache_miss', 'miss', 'percentage-of-absolute-row'],
50
- ['cache_hitpass', 'hitpass', 'percentage-of-absolute-row']]
51
- },
52
- 'current_poll_hit_rate': {
53
- 'options': [None, 'Current Poll Hit Rate Ratio', 'percentage', 'cache performance',
54
- 'varnish.current_poll_hit_rate', 'stacked'],
55
- 'lines': [
56
- ['cache_hit', 'hit', 'percentage-of-incremental-row'],
57
- ['cache_miss', 'miss', 'percentage-of-incremental-row'],
58
- ['cache_hitpass', 'hitpass', 'percentage-of-incremental-row']
59
- ]
60
- },
61
- 'cached_objects_expired': {
62
- 'options': [None, 'Expired Objects', 'expired/s', 'cache performance',
63
- 'varnish.cached_objects_expired', 'line'],
64
- 'lines': [
65
- ['n_expired', 'objects', 'incremental']
66
- ]
67
- },
68
- 'cached_objects_nuked': {
69
- 'options': [None, 'Least Recently Used Nuked Objects', 'nuked/s', 'cache performance',
70
- 'varnish.cached_objects_nuked', 'line'],
71
- 'lines': [
72
- ['n_lru_nuked', 'objects', 'incremental']
73
- ]
74
- },
75
- 'threads_total': {
76
- 'options': [None, 'Number Of Threads In All Pools', 'number', 'thread related metrics',
77
- 'varnish.threads_total', 'line'],
78
- 'lines': [
79
- ['threads', None, 'absolute']
80
- ]
81
- },
82
- 'threads_statistics': {
83
- 'options': [None, 'Threads Statistics', 'threads/s', 'thread related metrics',
84
- 'varnish.threads_statistics', 'line'],
85
- 'lines': [
86
- ['threads_created', 'created', 'incremental'],
87
- ['threads_failed', 'failed', 'incremental'],
88
- ['threads_limited', 'limited', 'incremental']
89
- ]
90
- },
91
- 'threads_queue_len': {
92
- 'options': [None, 'Current Queue Length', 'requests', 'thread related metrics',
93
- 'varnish.threads_queue_len', 'line'],
94
- 'lines': [
95
- ['thread_queue_len', 'in queue']
96
- ]
97
- },
98
- 'backend_connections': {
99
- 'options': [None, 'Backend Connections Statistics', 'connections/s', 'backend metrics',
100
- 'varnish.backend_connections', 'line'],
101
- 'lines': [
102
- ['backend_conn', 'successful', 'incremental'],
103
- ['backend_unhealthy', 'unhealthy', 'incremental'],
104
- ['backend_reuse', 'reused', 'incremental'],
105
- ['backend_toolate', 'closed', 'incremental'],
106
- ['backend_recycle', 'recycled', 'incremental'],
107
- ['backend_fail', 'failed', 'incremental']
108
- ]
109
- },
110
- 'backend_requests': {
111
- 'options': [None, 'Requests To The Backend', 'requests/s', 'backend metrics',
112
- 'varnish.backend_requests', 'line'],
113
- 'lines': [
114
- ['backend_req', 'sent', 'incremental']
115
- ]
116
- },
117
- 'esi_statistics': {
118
- 'options': [None, 'ESI Statistics', 'problems/s', 'esi related metrics', 'varnish.esi_statistics', 'line'],
119
- 'lines': [
120
- ['esi_errors', 'errors', 'incremental'],
121
- ['esi_warnings', 'warnings', 'incremental']
122
- ]
123
- },
124
- 'memory_usage': {
125
- 'options': [None, 'Memory Usage', 'MiB', 'memory usage', 'varnish.memory_usage', 'stacked'],
126
- 'lines': [
127
- ['memory_free', 'free', 'absolute', 1, 1 << 20],
128
- ['memory_allocated', 'allocated', 'absolute', 1, 1 << 20]]
129
- },
130
- 'uptime': {
131
- 'lines': [
132
- ['uptime', None, 'absolute']
133
- ],
134
- 'options': [None, 'Uptime', 'seconds', 'uptime', 'varnish.uptime', 'line']
135
- }
136
-}
137
-
138
-
139
-def backend_charts_template(name):
140
- order = [
141
- '{0}_response_statistics'.format(name),
142
- ]
143
-
144
- charts = {
145
- order[0]: {
146
- 'options': [None, 'Backend "{0}"'.format(name), 'kilobits/s', 'backend response statistics',
147
- 'varnish.backend', 'area'],
148
- 'lines': [
149
- ['{0}_beresp_hdrbytes'.format(name), 'header', 'incremental', 8, 1000],
150
- ['{0}_beresp_bodybytes'.format(name), 'body', 'incremental', -8, 1000]
151
- ]
152
- },
153
- }
154
-
155
- return order, charts
156
-
157
-
158
-def storage_charts_template(name):
159
- order = [
160
- 'storage_{0}_usage'.format(name),
161
- 'storage_{0}_alloc_objs'.format(name)
162
- ]
163
-
164
- charts = {
165
- order[0]: {
166
- 'options': [None, 'Storage "{0}" Usage'.format(name), 'KiB', 'storage usage', 'varnish.storage_usage', 'stacked'],
167
- 'lines': [
168
- ['{0}.g_space'.format(name), 'free', 'absolute', 1, 1 << 10],
169
- ['{0}.g_bytes'.format(name), 'allocated', 'absolute', 1, 1 << 10]
170
- ]
171
- },
172
- order[1]: {
173
- 'options': [None, 'Storage "{0}" Allocated Objects'.format(name), 'objects', 'storage usage', 'varnish.storage_alloc_objs', 'line'],
174
- 'lines': [
175
- ['{0}.g_alloc'.format(name), 'allocated', 'absolute']
176
- ]
177
- }
178
- }
179
-
180
- return order, charts
181
-
182
-
183
-VARNISHSTAT = 'varnishstat'
184
-
185
-re_version = re.compile(r'varnish-(?:plus-)?(?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)')
186
-
187
-
188
-class VarnishVersion:
189
- def __init__(self, major, minor, patch):
190
- self.major = major
191
- self.minor = minor
192
- self.patch = patch
193
-
194
- def __str__(self):
195
- return '{0}.{1}.{2}'.format(self.major, self.minor, self.patch)
196
-
197
-
198
-class Parser:
199
- _backend_new = re.compile(r'VBE.([\d\w_.]+)\(.*?\).(beresp[\w_]+)\s+(\d+)')
200
- _backend_old = re.compile(r'VBE\.[\d\w-]+\.([\w\d_-]+).(beresp[\w_]+)\s+(\d+)')
201
- _default = re.compile(r'([A-Z]+\.)?([\d\w_.]+)\s+(\d+)')
202
-
203
- def __init__(self):
204
- self.re_default = None
205
- self.re_backend = None
206
-
207
- def init(self, data):
208
- data = ''.join(data)
209
- parsed_main = Parser._default.findall(data)
210
- if parsed_main:
211
- self.re_default = Parser._default
212
-
213
- parsed_backend = Parser._backend_new.findall(data)
214
- if parsed_backend:
215
- self.re_backend = Parser._backend_new
216
- else:
217
- parsed_backend = Parser._backend_old.findall(data)
218
- if parsed_backend:
219
- self.re_backend = Parser._backend_old
220
-
221
- def server_stats(self, data):
222
- return self.re_default.findall(''.join(data))
223
-
224
- def backend_stats(self, data):
225
- return self.re_backend.findall(''.join(data))
226
-
227
-
228
-class Service(ExecutableService):
229
- def __init__(self, configuration=None, name=None):
230
- ExecutableService.__init__(self, configuration=configuration, name=name)
231
- self.order = ORDER
232
- self.definitions = CHARTS
233
- self.instance_name = configuration.get('instance_name')
234
- self.parser = Parser()
235
- self.command = None
236
- self.collected_vbe = set()
237
- self.collected_storages = set()
238
-
239
- def create_command(self):
240
- varnishstat = find_binary(VARNISHSTAT)
241
-
242
- if not varnishstat:
243
- self.error("can't locate '{0}' binary or binary is not executable by user netdata".format(VARNISHSTAT))
244
- return False
245
-
246
- command = [varnishstat, '-V']
247
- reply = self._get_raw_data(stderr=True, command=command)
248
- if not reply:
249
- self.error(
250
- "no output from '{0}'. Is varnish running? Not enough privileges?".format(' '.join(self.command)))
251
- return False
252
-
253
- ver = parse_varnish_version(reply)
254
- if not ver:
255
- self.error("failed to parse reply from '{0}', used regex :'{1}', reply : {2}".format(
256
- ' '.join(command), re_version.pattern, reply))
257
- return False
258
-
259
- if self.instance_name:
260
- self.command = [varnishstat, '-1', '-n', self.instance_name]
261
- else:
262
- self.command = [varnishstat, '-1']
263
-
264
- if ver.major > 4:
265
- self.command.extend(['-t', '1'])
266
-
267
- self.info("varnish version: {0}, will use command: '{1}'".format(ver, ' '.join(self.command)))
268
-
269
- return True
270
-
271
- def check(self):
272
- if not self.create_command():
273
- return False
274
-
275
- # STDOUT is not empty
276
- reply = self._get_raw_data()
277
- if not reply:
278
- self.error("no output from '{0}'. Is it running? Not enough privileges?".format(' '.join(self.command)))
279
- return False
280
-
281
- self.parser.init(reply)
282
-
283
- # Output is parsable
284
- if not self.parser.re_default:
285
- self.error('cant parse the output...')
286
- return False
287
-
288
- return True
289
-
290
- def get_data(self):
291
- """
292
- Format data received from shell command
293
- :return: dict
294
- """
295
- raw = self._get_raw_data()
296
- if not raw:
297
- return None
298
-
299
- data = dict()
300
- server_stats = self.parser.server_stats(raw)
301
- if not server_stats:
302
- return None
303
-
304
- stats = dict((param, value) for _, param, value in server_stats)
305
- data.update(stats)
306
-
307
- self.get_vbe_backends(data, raw)
308
- self.get_storages(server_stats)
309
-
310
- # varnish 5 uses default.g_bytes and default.g_space
311
- data['memory_allocated'] = data.get('s0.g_bytes') or data.get('default.g_bytes')
312
- data['memory_free'] = data.get('s0.g_space') or data.get('default.g_space')
313
-
314
- return data
315
-
316
- def get_vbe_backends(self, data, raw):
317
- if not self.parser.re_backend:
318
- return
319
- stats = self.parser.backend_stats(raw)
320
- if not stats:
321
- return
322
-
323
- for (name, param, value) in stats:
324
- data['_'.join([name, param])] = value
325
- if name in self.collected_vbe:
326
- continue
327
- self.collected_vbe.add(name)
328
- self.add_backend_charts(name)
329
-
330
- def get_storages(self, server_stats):
331
- # Storage types:
332
- # - SMF: File Storage
333
- # - SMA: Malloc Storage
334
- # - MSE: Massive Storage Engine (Varnish-Plus only)
335
- #
336
- # Stats example:
337
- # [('SMF.', 'ssdStorage.c_req', '47686'),
338
- # ('SMF.', 'ssdStorage.c_fail', '0'),
339
- # ('SMF.', 'ssdStorage.c_bytes', '668102656'),
340
- # ('SMF.', 'ssdStorage.c_freed', '140980224'),
341
- # ('SMF.', 'ssdStorage.g_alloc', '39753'),
342
- # ('SMF.', 'ssdStorage.g_bytes', '527122432'),
343
- # ('SMF.', 'ssdStorage.g_space', '53159968768'),
344
- # ('SMF.', 'ssdStorage.g_smf', '40130'),
345
- # ('SMF.', 'ssdStorage.g_smf_frag', '311'),
346
- # ('SMF.', 'ssdStorage.g_smf_large', '66')]
347
- storages = [name for typ, name, _ in server_stats if typ.startswith(('SMF', 'SMA', 'MSE')) and name.endswith('g_space')]
348
- if not storages:
349
- return
350
- for storage in storages:
351
- storage = storage.split('.')[0]
352
- if storage in self.collected_storages:
353
- continue
354
- self.collected_storages.add(storage)
355
- self.add_storage_charts(storage)
356
-
357
- def add_backend_charts(self, backend_name):
358
- self.add_charts(backend_name, backend_charts_template)
359
-
360
- def add_storage_charts(self, storage_name):
361
- self.add_charts(storage_name, storage_charts_template)
362
-
363
- def add_charts(self, name, charts_template):
364
- order, charts = charts_template(name)
365
-
366
- for chart_name in order:
367
- params = [chart_name] + charts[chart_name]['options']
368
- dimensions = charts[chart_name]['lines']
369
-
370
- new_chart = self.charts.add_chart(params)
371
- for dimension in dimensions:
372
- new_chart.add_dimension(dimension)
373
-
374
-
375
-def parse_varnish_version(lines):
376
- m = re_version.search(lines[0])
377
- if not m:
378
- return None
379
-
380
- m = m.groupdict()
381
- return VarnishVersion(
382
- int(m['major']),
383
- int(m['minor']),
384
- int(m['patch']),
385
- )
src/collectors/python.d.plugin/varnish/varnish.conf
deleted
-66
@@ -1,66 +0,0 @@
1
-# netdata python.d.plugin configuration for varnish
2
-#
3
-# This file is in YaML format. Generally the format is:
4
-#
5
-# name: value
6
-#
7
-# There are 2 sections:
8
-# - global variables
9
-# - one or more JOBS
10
-#
11
-# JOBS allow you to collect values from multiple sources.
12
-# Each source will have its own set of charts.
13
-#
14
-# JOB parameters have to be indented (using spaces only, example below).
15
-
16
-# ----------------------------------------------------------------------
17
-# Global Variables
18
-# These variables set the defaults for all JOBs, however each JOB
19
-# may define its own, overriding the defaults.
20
-
21
-# update_every sets the default data collection frequency.
22
-# If unset, the python.d.plugin default is used.
23
-# update_every: 1
24
-
25
-# priority controls the order of charts at the netdata dashboard.
26
-# Lower numbers move the charts towards the top of the page.
27
-# If unset, the default for python.d.plugin is used.
28
-# priority: 60000
29
-
30
-# penalty indicates whether to apply penalty to update_every in case of failures.
31
-# Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes.
32
-# penalty: yes
33
-
34
-# autodetection_retry sets the job re-check interval in seconds.
35
-# The job is not deleted if check fails.
36
-# Attempts to start the job are made once every autodetection_retry.
37
-# This feature is disabled by default.
38
-# autodetection_retry: 0
39
-
40
-# ----------------------------------------------------------------------
41
-# JOBS (data collection sources)
42
-#
43
-# The default JOBS share the same *name*. JOBS with the same name
44
-# are mutually exclusive. Only one of them will be allowed running at
45
-# any time. This allows autodetection to try several alternatives and
46
-# pick the one that works.
47
-#
48
-# Any number of jobs is supported.
49
-#
50
-# All python.d.plugin JOBS (for all its modules) support a set of
51
-# predefined parameters. These are:
52
-#
53
-# job_name:
54
-# name: myname # the JOB's name as it will appear at the
55
-# # dashboard (by default is the job_name)
56
-# # JOBs sharing a name are mutually exclusive
57
-# update_every: 1 # the JOB's data collection frequency
58
-# priority: 60000 # the JOB's order on the dashboard
59
-# penalty: yes # the JOB's penalty
60
-# autodetection_retry: 0 # the JOB's re-check interval in seconds
61
-#
62
-# Additionally to the above, varnish also supports the following:
63
-#
64
-# instance_name: 'name' # the name of the varnishd instance to get logs from. If not specified, the host name is used.
65
-#
66
-# ----------------------------------------------------------------------