Reorg markdown files for learn (#14547)
* Move export graphite metrics guide content to new integrations landing page and exporter readme * Merge info on how to write python collector and reorg file * Codacy warning fixes
Chris Akritidis committed
Feb 16, 2023 at 13:28 UTC
3e507768b9ef255eae869403172dc17e32a821d0
5 files changed
+315
-424
collectors/python.d.plugin/README.md
+1
-195
@@ -74,198 +74,4 @@ Where `[module]` is the directory name under <https://github.com/netdata/netdata
74
75
## How to write a new module
76
77
-Writing new python module is simple. You just need to remember to include 5 major things:
78
-
79
-- **ORDER** global list
80
-- **CHART** global dictionary
81
-- **Service** class
82
-- **\_get_data** method
83
-
84
-If you plan to submit the module in a PR, make sure and go through the [PR checklist for new modules](#pull-request-checklist-for-python-plugins) beforehand to make sure you have updated all the files you need to.
85
-
86
-For a quick start, you can look at the [example
87
-plugin](https://raw.githubusercontent.com/netdata/netdata/master/collectors/python.d.plugin/example/example.chart.py).
88
-
89
-**Note**: If you are working 'locally' on a new collector and would like to run it in an already installed and running
90
-Netdata (as opposed to having to install Netdata from source again with your new changes) to can copy over the relevant
91
-file to where Netdata expects it and then either `sudo systemctl restart netdata` to have it be picked up and used by
92
-Netdata or you can just run the updated collector in debug mode by following a process like below (this assumes you have
93
-[installed Netdata from a GitHub fork](https://github.com/netdata/netdata/blob/master/packaging/installer/methods/manual.md) you
94
-have made to do your development on).
95
-
96
-```bash
97
-# clone your fork (done once at the start but shown here for clarity)
98
-#git clone --branch my-example-collector https://github.com/mygithubusername/netdata.git --depth=100 --recursive
99
-# go into your netdata source folder
100
-cd netdata
101
-# git pull your latest changes (assuming you built from a fork you are using to develop on)
102
-git pull
103
-# instead of running the installer we can just copy over the updated collector files
104
-#sudo ./netdata-installer.sh --dont-wait
105
-# copy over the file you have updated locally (pretending we are working on the 'example' collector)
106
-sudo cp collectors/python.d.plugin/example/example.chart.py /usr/libexec/netdata/python.d/
107
-# become user netdata
108
-sudo su -s /bin/bash netdata
109
-# run your updated collector in debug mode to see if it works without having to reinstall netdata
110
-/usr/libexec/netdata/plugins.d/python.d.plugin example debug trace nolock
111
-```
112
-
113
-### Global variables `ORDER` and `CHART`
114
-
115
-`ORDER` list should contain the order of chart ids. Example:
116
-
117
-```py
118
-ORDER = ['first_chart', 'second_chart', 'third_chart']
119
-```
120
-
121
-`CHART` dictionary is a little bit trickier. It should contain the chart definition in following format:
122
-
123
-```py
124
-CHART = {
125
- id: {
126
- 'options': [name, title, units, family, context, charttype],
127
- 'lines': [
128
- [unique_dimension_name, name, algorithm, multiplier, divisor]
129
- ]}
130
-```
131
-
132
-All names are better explained in the [External Plugins](https://github.com/netdata/netdata/blob/master/collectors/plugins.d/README.md) section.
133
-Parameters like `priority` and `update_every` are handled by `python.d.plugin`.
134
-
135
-### `Service` class
136
-
137
-Every module needs to implement its own `Service` class. This class should inherit from one of the framework classes:
138
-
139
-- `SimpleService`
140
-- `UrlService`
141
-- `SocketService`
142
-- `LogService`
143
-- `ExecutableService`
144
-
145
-Also it needs to invoke the parent class constructor in a specific way as well as assign global variables to class variables.
146
-
147
-Simple example:
148
-
149
-```py
150
-from base import UrlService
151
-class Service(UrlService):
152
- def __init__(self, configuration=None, name=None):
153
- UrlService.__init__(self, configuration=configuration, name=name)
154
- self.order = ORDER
155
- self.definitions = CHARTS
156
-```
157
-
158
-### `_get_data` collector/parser
159
-
160
-This method should grab raw data from `_get_raw_data`, parse it, and return a dictionary where keys are unique dimension names or `None` if no data is collected.
161
-
162
-Example:
163
-
164
-```py
165
-def _get_data(self):
166
- try:
167
- raw = self._get_raw_data().split(" ")
168
- return {'active': int(raw[2])}
169
- except (ValueError, AttributeError):
170
- return None
171
-```
172
-
173
-# More about framework classes
174
-
175
-Every framework class has some user-configurable variables which are specific to this particular class. Those variables should have default values initialized in the child class constructor.
176
-
177
-If module needs some additional user-configurable variable, it can be accessed from the `self.configuration` list and assigned in constructor or custom `check` method. Example:
178
-
179
-```py
180
-def __init__(self, configuration=None, name=None):
181
- UrlService.__init__(self, configuration=configuration, name=name)
182
- try:
183
- self.baseurl = str(self.configuration['baseurl'])
184
- except (KeyError, TypeError):
185
- self.baseurl = "http://localhost:5001"
186
-```
187
-
188
-Classes implement `_get_raw_data` which should be used to grab raw data. This method usually returns a list of strings.
189
-
190
-### `SimpleService`
191
-
192
-_This is last resort class, if a new module cannot be written by using other framework class this one can be used._
193
-
194
-_Example: `ceph`, `sensors`_
195
-
196
-It is the lowest-level class which implements most of module logic, like:
197
-
198
-- threading
199
-- handling run times
200
-- chart formatting
201
-- logging
202
-- chart creation and updating
203
-
204
-### `LogService`
205
-
206
-_Examples: `apache_cache`, `nginx_log`_
207
-
208
-_Variable from config file_: `log_path`.
209
-
210
-Object created from this class reads new lines from file specified in `log_path` variable. It will check if file exists and is readable. Also `_get_raw_data` returns list of strings where each string is one line from file specified in `log_path`.
211
-
212
-### `ExecutableService`
213
-
214
-_Examples: `exim`, `postfix`_
215
-
216
-_Variable from config file_: `command`.
217
-
218
-This allows to execute a shell command in a secure way. It will check for invalid characters in `command` variable and won't proceed if there is one of:
219
-
220
-- '&'
221
-- '|'
222
-- ';'
223
-- '>'
224
-- '\<'
225
-
226
-For additional security it uses python `subprocess.Popen` (without `shell=True` option) to execute command. Command can be specified with absolute or relative name. When using relative name, it will try to find `command` in `PATH` environment variable as well as in `/sbin` and `/usr/sbin`.
227
-
228
-`_get_raw_data` returns list of decoded lines returned by `command`.
229
-
230
-### UrlService
231
-
232
-_Examples: `apache`, `nginx`, `tomcat`_
233
-
234
-_Variables from config file_: `url`, `user`, `pass`.
235
-
236
-If data is grabbed by accessing service via HTTP protocol, this class can be used. It can handle HTTP Basic Auth when specified with `user` and `pass` credentials.
237
-
238
-Please note that the config file can use different variables according to the specification of each module.
239
-
240
-`_get_raw_data` returns list of utf-8 decoded strings (lines).
241
-
242
-### SocketService
243
-
244
-_Examples: `dovecot`, `redis`_
245
-
246
-_Variables from config file_: `unix_socket`, `host`, `port`, `request`.
247
-
248
-Object will try execute `request` using either `unix_socket` or TCP/IP socket with combination of `host` and `port`. This can access unix sockets with SOCK_STREAM or SOCK_DGRAM protocols and TCP/IP sockets in version 4 and 6 with SOCK_STREAM setting.
249
-
250
-Sockets are accessed in non-blocking mode with 15 second timeout.
251
-
252
-After every execution of `_get_raw_data` socket is closed, to prevent this module needs to set `_keep_alive` variable to `True` and implement custom `_check_raw_data` method.
253
-
254
-`_check_raw_data` should take raw data and return `True` if all data is received otherwise it should return `False`. Also it should do it in fast and efficient way.
255
-
256
-## Pull Request Checklist for Python Plugins
257
-
258
-This is a generic checklist for submitting a new Python plugin for Netdata. It is by no means comprehensive.
259
-
260
-At minimum, to be buildable and testable, the PR needs to include:
261
-
262
-- The module itself, following proper naming conventions: `collectors/python.d.plugin/<module_dir>/<module_name>.chart.py`
263
-- A README.md file for the plugin under `collectors/python.d.plugin/<module_dir>`.
264
-- The configuration file for the module: `collectors/python.d.plugin/<module_dir>/<module_name>.conf`. Python config files are in YAML format, and should include comments describing what options are present. The instructions are also needed in the configuration section of the README.md
265
-- A basic configuration for the plugin in the appropriate global config file: `collectors/python.d.plugin/python.d.conf`, which is also in YAML format. Either add a line that reads `# <module_name>: yes` if the module is to be enabled by default, or one that reads `<module_name>: no` if it is to be disabled by default.
266
-- A makefile for the plugin at `collectors/python.d.plugin/<module_dir>/Makefile.inc`. Check an existing plugin for what this should look like.
267
-- A line in `collectors/python.d.plugin/Makefile.am` including the above-mentioned makefile. Place it with the other plugin includes (please keep the includes sorted alphabetically).
268
-- Optionally, chart information in `web/gui/dashboard_info.js`. This generally involves specifying a name and icon for the section, and may include descriptions for the section or individual charts.
269
-- Optionally, some default alarm configurations for your collector in `health/health.d/<module_name>.conf` and a line adding `<module_name>.conf` in `health/Makefile.am`.
270
-
271
-
77
+See [develop a custom collector in Python](https://github.com/netdata/netdata/edit/master/docs/guides/python-collector.md).
docs/category-overview-pages/integrations-overview.md
new
+30
@@ -0,0 +1,30 @@
1
+<!--
2
+title: "Integrations"
3
+sidebar_label: "Integrations"
4
+custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/category-overview-pages/integrations-overview.md"
5
+description: "Available integrations in Netdata"
6
+learn_status: "Published"
7
+learn_rel_path: "Integrations"
8
+-->
9
+
10
+# Netdata Integrations
11
+
12
+Netdata's ability to monitor out of the box every potentially useful aspect of a node's operation is unparalleled.
13
+But Netdata also provides out of the box, meaningful charts and alerts for hundreds of applications, with the ability
14
+to be easily extended to monitor anything. See the full list of Netdata's capabilities and how you can extend them in the
15
+[supported collectors list](https://github.com/netdata/netdata/blobl/master/collectors/COLLECTORS.md).
16
+
17
+Our out of the box alerts were created by expert professionals and have been validated on the field, countless times.
18
+Use them to trigger [alert notifications](https://github.com/netdata/netdata/blob/master/docs/monitor/enable-notifications.md)
19
+either centrally, via the
20
+[Cloud alert notifications](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/notifications.md)
21
+, or by configuring individual
22
+[agent notifications](https://github.com/netdata/netdata/blobl/master/health/notifications/README.md).
23
+
24
+We designed Netdata with interoperability in mind. The Agent collects thousands of metrics every second, and then what
25
+you do with them is up to you. You can
26
+[store metrics in the database engine](https://github.com/netdata/netdata/blob/master/docs/guides/longer-metrics-storage.md),
27
+or send them to another time series database for long-term storage or further analysis using
28
+Netdata's [exporting engine](https://github.com/netdata/netdata/edit/master/exporting/README.md).
29
+
30
+
docs/guides/export/export-netdata-metrics-graphite.md
deleted
-185
@@ -1,185 +0,0 @@
1
-<!--
2
-title: "Export and visualize Netdata metrics in Graphite"
3
-sidebar_label: "Export and visualize Netdata metrics in Graphite"
4
-custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/guides/export/export-netdata-metrics-graphite.md"
5
-description: "Use Netdata to collect and export thousands of metrics to Graphite for long-term storage or further analysis."
6
-image: /img/seo/guides/export/export-netdata-metrics-graphite.png
7
-learn_status: "Published"
8
-learn_topic_type: "Tasks"
9
-learn_rel_path: "Guides"
10
--->
11
-import { OneLineInstallWget } from '@site/src/components/OneLineInstall/'
12
-
13
-# Export and visualize Netdata metrics in Graphite
14
-
15
-Collecting metrics is an essential part of monitoring any application, service, or infrastructure, but it's not the
16
-final step for any developer, sysadmin, SRE, or DevOps engineer who's keeping an eye on things. To take meaningful
17
-action on these metrics, you may need to develop a stack of monitoring tools that work in parallel to help you diagnose
18
-anomalies and discover root causes faster.
19
-
20
-We designed Netdata with interoperability in mind. The Agent collects thousands of metrics every second, and then what
21
-you do with them is up to you. You
22
-can [store metrics in the database engine](https://github.com/netdata/netdata/blob/master/docs/guides/longer-metrics-storage.md),
23
-or send them to another time series database for long-term storage or further analysis using
24
-Netdata's [exporting engine](https://github.com/netdata/netdata/blob/master/docs/export/external-databases.md).
25
-
26
-In this guide, we'll show you how to export Netdata metrics to [Graphite](https://graphiteapp.org/) for long-term
27
-storage and further analysis. Graphite is a free open-source software (FOSS) tool that collects graphs numeric
28
-time-series data, such as all the metrics collected by the Netdata Agent itself. Using Netdata and Graphite together,
29
-you get more visibility into the health and performance of your entire infrastructure.
30
-
31
-
33
-
34
-Let's get started.
35
-
36
-## Install the Netdata Agent
37
-
38
-If you don't have the Netdata Agent installed already, visit
39
-the [installation guide](https://github.com/netdata/netdata/blob/master/packaging/installer/README.md)
40
-for the recommended instructions for your system. In most cases, you can use the one-line installation script:
41
-
42
-<OneLineInstallWget/>
43
-
44
-Once installation finishes, open your browser and navigate to `http://NODE:19999`, replacing `NODE` with the IP address
45
-or hostname of your system, to find the Agent dashboard.
46
-
47
-## Install Graphite via Docker
48
-
49
-For this guide, we'll install Graphite using Docker. See the [Docker documentation](https://docs.docker.com/get-docker/)
50
-for details if you don't yet have it installed on your system.
51
-
52
-> If you already have Graphite installed, skip this step. If you want to install via a different method, see the
53
-> [Graphite installation docs](https://graphite.readthedocs.io/en/latest/install.html), with the caveat that some
54
-> configuration settings may be different.
55
-
56
-Start up the Graphite image with `docker run`.
57
-
58
-```bash
59
-docker run -d \
60
- --name graphite \
61
- --restart=always \
62
- -p 80:80 \
63
- -p 2003-2004:2003-2004 \
64
- -p 2023-2024:2023-2024 \
65
- -p 8125:8125/udp \
66
- -p 8126:8126 \
67
- graphiteapp/graphite-statsd
68
-```
69
-
70
-Open your browser and navigate to `http://NODE`, to see the Graphite interface. Nothing yet, but we'll fix that soon
71
-enough.
72
-
73
-
74
-
75
-## Enable the Graphite exporting connector
76
-
77
-You're now ready to begin exporting Netdata metrics to Graphite.
78
-
79
-Begin by using `edit-config` to open the `exporting.conf` file.
80
-
81
-```bash
82
-cd /etc/netdata # Replace this path with your Netdata config directory
83
-sudo ./edit-config exporting.conf
84
-```
85
-
86
-If you haven't already, enable the exporting engine by setting `enabled` to `yes` in the `[exporting:global]` section.
87
-
88
-```conf
89
-[exporting:global]
90
- enabled = yes
91
-```
92
-
93
-Next, configure the connector. Find the `[graphite:my_graphite_instance]` example section and uncomment the line.
94
-Replace `my_graphite_instance` with a name of your choice. Let's go with `[graphite:netdata]`. Set `enabled` to `yes`
95
-and uncomment the line. Your configuration should now look like this:
96
-
97
-```conf
98
-[graphite:netdata]
99
- enabled = yes
100
- # destination = localhost
101
- # data source = average
102
- # prefix = netdata
103
- # hostname = my_hostname
104
- # update every = 10
105
- # buffer on failures = 10
106
- # timeout ms = 20000
107
- # send names instead of ids = yes
108
- # send charts matching = *
109
- # send hosts matching = localhost *
110
-```
111
-
112
-Set the `destination` setting to `localhost:2003`. By default, the Docker image for Graphite listens on port `2003` for
113
-incoming metrics. If you installed Graphite a different way, or tweaked the `docker run` command, you may need to change
114
-the port accordingly.
115
-
116
-```conf
117
-[graphite:netdata]
118
- enabled = yes
119
- destination = localhost:2003
120
- ...
121
-```
122
-
123
-We'll not worry about the rest of the settings for now. Restart the Agent using `sudo systemctl restart netdata`, or the
124
-[appropriate method](https://github.com/netdata/netdata/blob/master/docs/configure/start-stop-restart.md) for your
125
-system, to spin up the exporting engine.
126
-
127
-## See and organize Netdata metrics in Graphite
128
-
129
-Head back to the Graphite interface again, then click on the **Dashboard** link to get started with Netdata's exported
130
-metrics. You can also navigate directly to `http://NODE/dashboard`.
131
-
132
-Let's switch the interface to help you understand which metrics Netdata is exporting to Graphite. Click on **Dashboard**
133
-and **Configure UI**, then choose the **Tree** option. Refresh your browser to change the UI.
134
-
135
-
136
-
137
-You should now see a tree of available contexts, including one that matches the hostname of the Agent exporting metrics.
138
-In this example, the Agent's hostname is `arcturus`.
139
-
140
-Let's add some system CPU charts so you can monitor the long-term health of your system. Click through the tree to find
141
-**hostname → system → cpu** metrics, then click on the **user** context. A chart with metrics from that context appears
142
-in the dashboard. Add a few other system CPU charts to flesh things out.
143
-
144
-Next, let's combine one or two of these charts. Click and drag one chart onto the other, and wait until the green **Drop
145
-to merge** dialog appears. Release to merge the charts.
146
-
147
-
148
-
149
-Finally, save your dashboard. Click **Dashboard**, then **Save As**, then choose a name. Your dashboard is now saved.
150
-
151
-Of course, this is just the beginning of the customization you can do with Graphite. You can change the time range,
152
-share your dashboard with others, or use the composer to customize the size and appearance of specific charts. Learn
153
-more about adding, modifying, and combining graphs in
154
-the [Graphite docs](https://graphite.readthedocs.io/en/latest/dashboard.html).
155
-
156
-## Monitor the exporting engine
157
-
158
-As soon as the exporting engine begins, Netdata begins reporting metrics about the system's health and performance.
159
-
160
-
161
-
162
-You can use these charts to verify that Netdata is properly exporting metrics to Graphite. You can even add these
163
-exporting charts to your Graphite dashboard!
164
-
165
-### Add exporting charts to Netdata Cloud
166
-
167
-You can also show these exporting engine metrics on Netdata Cloud. If you don't have an account already,
168
-go [sign in](https://app.netdata.cloud) and get started for free.
169
-
170
-Add more metrics to a War Room's Nodes view by clicking on the **Add metric** button, then typing `exporting` into the
171
-context field. Choose the exporting contexts you want to add, then click **Add**. You'll see these charts alongside any
172
-others you've customized in Netdata Cloud.
173
-
174
-
175
-
176
-## What's next?
177
-
178
-What you do with your exported metrics is entirely up to you, but as you might have seen in the Graphite connector
179
-configuration block, there are many other ways to tweak and customize which metrics you export to Graphite and how
180
-often.
181
-
182
-For full details about each configuration option and what it does, see
183
-the [exporting reference guide](https://github.com/netdata/netdata/blob/master/exporting/README.md).
184
-
185
-
docs/guides/python-collector.md
+181
-39
@@ -8,32 +8,58 @@ author_title: "University of Patras"
8
author_img: "/img/authors/panagiotis-papaioannou.jpg"
9
custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/guides/python-collector.md
10
learn_status: "Published"
11
-learn_topic_type: "Tasks"
12
-learn_rel_path: "Guides"
11
+learn_rel_path: "Developers/External plugins/python.d.plugin"
12
-->
13
14
# Develop a custom data collector in Python
15
17
-The Netdata Agent uses [data collectors](https://github.com/netdata/netdata/blob/master/collectors/README.md) to fetch metrics from hundreds of system,
18
-container, and service endpoints. While the Netdata team and community has built [powerful
19
-collectors](https://github.com/netdata/netdata/blob/master/collectors/COLLECTORS.md) for most system, container, and service/application endpoints, there are plenty
20
-of custom applications that can't be monitored by default.
21
-
22
-## Problem
23
-
24
-You have a custom application or infrastructure that you need to monitor, but no open-source monitoring tool offers a
25
-prebuilt method for collecting your required metric data.
26
-
27
-## Solution
16
+The Netdata Agent uses [data collectors](https://github.com/netdata/netdata/blob/master/collectors/README.md) to
17
+fetch metrics from hundreds of system, container, and service endpoints. While the Netdata team and community has built
18
+[powerful collectors](https://github.com/netdata/netdata/blob/master/collectors/COLLECTORS.md) for most system, container,
19
+and service/application endpoints, some custom applications can't be monitored by default.
20
21
In this tutorial, you'll learn how to leverage the [Python programming language](https://www.python.org/) to build a
22
custom data collector for the Netdata Agent. Follow along with your own dataset, using the techniques and best practices
23
covered here, or use the included examples for collecting and organizing either random or weather data.
24
25
+If you're comfortable with Golang, consider instead writing a module for the [go.d.plugin](https://github.com/netdata/go.d.plugin).
26
+Golang is more performant, easier to maintain, and simpler for users since it doesn't require a particular runtime on the node to
27
+execute. Python plugins require Python on the machine to be executed. Netdata uses Go as the platform of choice for
28
+production-grade collectors.
29
+
30
## What you need to get started
31
35
-- A physical or virtual Linux system, which we'll call a _node_.
36
-- A working installation of the free and open-source [Netdata](https://github.com/netdata/netdata/blob/master/packaging/installer/README.md) monitoring agent.
32
+ - A physical or virtual Linux system, which we'll call a _node_.
33
+ - A working [installation of Netdata](https://github.com/netdata/netdata/blob/master/packaging/installer/README.md) monitoring agent.
34
+
35
+### Quick start
36
+
37
+For a quick start, you can look at the
38
+[example plugin](https://raw.githubusercontent.com/netdata/netdata/master/collectors/python.d.plugin/example/example.chart.py).
39
+
40
+**Note**: If you are working 'locally' on a new collector and would like to run it in an already installed and running
41
+Netdata (as opposed to having to install Netdata from source again with your new changes) you can copy over the relevant
42
+file to where Netdata expects it and then either `sudo systemctl restart netdata` to have it be picked up and used by
43
+Netdata or you can just run the updated collector in debug mode by following a process like below (this assumes you have
44
+[installed Netdata from a GitHub fork](https://github.com/netdata/netdata/blob/master/packaging/installer/methods/manual.md) you
45
+have made to do your development on).
46
+
47
+```bash
48
+# clone your fork (done once at the start but shown here for clarity)
49
+#git clone --branch my-example-collector https://github.com/mygithubusername/netdata.git --depth=100 --recursive
50
+# go into your netdata source folder
51
+cd netdata
52
+# git pull your latest changes (assuming you built from a fork you are using to develop on)
53
+git pull
54
+# instead of running the installer we can just copy over the updated collector files
55
+#sudo ./netdata-installer.sh --dont-wait
56
+# copy over the file you have updated locally (pretending we are working on the 'example' collector)
57
+sudo cp collectors/python.d.plugin/example/example.chart.py /usr/libexec/netdata/python.d/
58
+# become user netdata
59
+sudo su -s /bin/bash netdata
60
+# run your updated collector in debug mode to see if it works without having to reinstall netdata
61
+/usr/libexec/netdata/plugins.d/python.d.plugin example debug trace nolock
62
+```
63
64
## Jobs and elements of a Python collector
65
@@ -54,6 +80,11 @@ The basic elements of a Netdata collector are:
80
- `data{}`: A dictionary containing the values to be displayed.
81
- `get_data()`: The basic function of the plugin which will return to Netdata the correct values.
82
83
+**Note**: All names are better explained in the
84
+[External Plugins Documentation](https://github.com/netdata/netdata/blob/master/collectors/plugins.d/README.md).
85
+Parameters like `priority` and `update_every` mentioned in that documentation are handled by the `python.d.plugin`,
86
+not by each collection module.
87
+
88
Let's walk through these jobs and elements as independent elements first, then apply them to example Python code.
89
90
### Determine how to gather metrics data
@@ -139,11 +170,18 @@ correct values.
170
171
## Framework classes
172
142
-The `python.d` plugin has a number of framework classes that can be used to speed up the development of your python
143
-collector. Your class can inherit one of these framework classes, which have preconfigured methods.
173
+Every module needs to implement its own `Service` class. This class should inherit from one of the framework classes:
174
+
175
+- `SimpleService`
176
+- `UrlService`
177
+- `SocketService`
178
+- `LogService`
179
+- `ExecutableService`
180
+
181
+Also it needs to invoke the parent class constructor in a specific way as well as assign global variables to class variables.
182
145
-For example, the snippet below is from the [RabbitMQ
146
-collector](https://github.com/netdata/netdata/blob/91f3268e9615edd393bd43de4ad8068111024cc9/collectors/python.d.plugin/rabbitmq/rabbitmq.chart.py#L273).
183
+For example, the snippet below is from the
184
+[RabbitMQ collector](https://github.com/netdata/netdata/blob/91f3268e9615edd393bd43de4ad8068111024cc9/collectors/python.d.plugin/rabbitmq/rabbitmq.chart.py#L273).
185
This collector uses an HTTP endpoint and uses the `UrlService` framework class, which only needs to define an HTTP
186
endpoint for data collection.
187
@@ -170,8 +208,7 @@ class Service(UrlService):
208
209
In our use-case, we use the `SimpleService` framework, since there is no framework class that suits our needs.
210
173
-You can read more about the [framework classes](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/README.md#how-to-write-a-new-module) from
174
-the Netdata documentation.
211
+You can find below the [framework class reference](#framework-class-reference).
212
213
## An example collector using weather station data
214
@@ -200,6 +237,35 @@ CHARTS = {
237
238
## Parse the data to extract or create the actual data to be represented
239
240
+Every collector must implement `_get_data`. This method should grab raw data from `_get_raw_data`,
241
+parse it, and return a dictionary where keys are unique dimension names, or `None` if no data is collected.
242
+
243
+For example:
244
+```py
245
+def _get_data(self):
246
+ try:
247
+ raw = self._get_raw_data().split(" ")
248
+ return {'active': int(raw[2])}
249
+ except (ValueError, AttributeError):
250
+ return None
251
+```
252
+
253
+In our weather data collector we declare `_get_data` as follows:
254
+
255
+```python
256
+ def get_data(self):
257
+ #The data dict is basically all the values to be represented
258
+ # The entries are in the format: { "dimension": value}
259
+ #And each "dimension" should belong to a chart.
260
+ data = dict()
261
+
262
+ self.populate_data()
263
+
264
+ data['current_temperature'] = self.weather_data["temp"]
265
+
266
+ return data
267
+```
268
+
269
A standard practice would be to either get the data on JSON format or transform them to JSON format. We use a dictionary
270
to give this format and issue random values to simulate received data.
271
@@ -465,26 +531,102 @@ variables and inform the user about the defaults. For example, take a look at th
531
You can read more about the configuration file on the [`python.d.plugin`
532
documentation](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/README.md).
533
468
-## What's next?
534
+You can find the source code for the above examples on [GitHub](https://github.com/papajohn-uop/netdata).
535
+
536
+## Pull Request Checklist for Python Plugins
537
+
538
+This is a generic checklist for submitting a new Python plugin for Netdata. It is by no means comprehensive.
539
+
540
+At minimum, to be buildable and testable, the PR needs to include:
541
+
542
+- The module itself, following proper naming conventions: `collectors/python.d.plugin/<module_dir>/<module_name>.chart.py`
543
+- A README.md file for the plugin under `collectors/python.d.plugin/<module_dir>`.
544
+- The configuration file for the module: `collectors/python.d.plugin/<module_dir>/<module_name>.conf`. Python config files are in YAML format, and should include comments describing what options are present. The instructions are also needed in the configuration section of the README.md
545
+- A basic configuration for the plugin in the appropriate global config file: `collectors/python.d.plugin/python.d.conf`, which is also in YAML format. Either add a line that reads `# <module_name>: yes` if the module is to be enabled by default, or one that reads `<module_name>: no` if it is to be disabled by default.
546
+- A makefile for the plugin at `collectors/python.d.plugin/<module_dir>/Makefile.inc`. Check an existing plugin for what this should look like.
547
+- A line in `collectors/python.d.plugin/Makefile.am` including the above-mentioned makefile. Place it with the other plugin includes (please keep the includes sorted alphabetically).
548
+- Optionally, chart information in `web/gui/dashboard_info.js`. This generally involves specifying a name and icon for the section, and may include descriptions for the section or individual charts.
549
+- Optionally, some default alarm configurations for your collector in `health/health.d/<module_name>.conf` and a line adding `<module_name>.conf` in `health/Makefile.am`.
550
+
551
+## Framework class reference
552
+
553
+Every framework class has some user-configurable variables which are specific to this particular class. Those variables should have default values initialized in the child class constructor.
554
+
555
+If module needs some additional user-configurable variable, it can be accessed from the `self.configuration` list and assigned in constructor or custom `check` method. Example:
556
+
557
+```py
558
+def __init__(self, configuration=None, name=None):
559
+ UrlService.__init__(self, configuration=configuration, name=name)
560
+ try:
561
+ self.baseurl = str(self.configuration['baseurl'])
562
+ except (KeyError, TypeError):
563
+ self.baseurl = "http://localhost:5001"
564
+```
565
+
566
+Classes implement `_get_raw_data` which should be used to grab raw data. This method usually returns a list of strings.
567
+
568
+### `SimpleService`
569
+
570
+This is last resort class, if a new module cannot be written by using other framework class this one can be used.
571
+
572
+Example: `ceph`, `sensors`
573
+
574
+It is the lowest-level class which implements most of module logic, like:
575
+
576
+- threading
577
+- handling run times
578
+- chart formatting
579
+- logging
580
+- chart creation and updating
581
+
582
+### `LogService`
583
+
584
+Examples: `apache_cache`, `nginx_log`_
585
+
586
+Variable from config file: `log_path`.
587
+
588
+Object created from this class reads new lines from file specified in `log_path` variable. It will check if file exists and is readable. Also `_get_raw_data` returns list of strings where each string is one line from file specified in `log_path`.
589
+
590
+### `ExecutableService`
591
+
592
+Examples: `exim`, `postfix`_
593
+
594
+Variable from config file: `command`.
595
+
596
+This allows to execute a shell command in a secure way. It will check for invalid characters in `command` variable and won't proceed if there is one of:
597
+
598
+- '&'
599
+- '|'
600
+- ';'
601
+- '>'
602
+- '\<'
603
+
604
+For additional security it uses python `subprocess.Popen` (without `shell=True` option) to execute command. Command can be specified with absolute or relative name. When using relative name, it will try to find `command` in `PATH` environment variable as well as in `/sbin` and `/usr/sbin`.
605
+
606
+`_get_raw_data` returns list of decoded lines returned by `command`.
607
+
608
+### UrlService
609
+
610
+Examples: `apache`, `nginx`, `tomcat`_
611
+
612
+Variables from config file: `url`, `user`, `pass`.
613
+
614
+If data is grabbed by accessing service via HTTP protocol, this class can be used. It can handle HTTP Basic Auth when specified with `user` and `pass` credentials.
615
+
616
+Please note that the config file can use different variables according to the specification of each module.
617
+
618
+`_get_raw_data` returns list of utf-8 decoded strings (lines).
619
+
620
+### SocketService
621
+
622
+Examples: `dovecot`, `redis`
623
470
-Find the source code for the above examples on [GitHub](https://github.com/papajohn-uop/netdata).
624
+Variables from config file: `unix_socket`, `host`, `port`, `request`.
625
472
-Now you are ready to start developing our Netdata python Collector and share it with the rest of the Netdata community.
626
+Object will try execute `request` using either `unix_socket` or TCP/IP socket with combination of `host` and `port`. This can access unix sockets with SOCK_STREAM or SOCK_DGRAM protocols and TCP/IP sockets in version 4 and 6 with SOCK_STREAM setting.
627
474
-- If you need help while developing your collector, join our [Netdata
475
- Community](https://community.netdata.cloud/c/agent-development/9) to chat about it.
476
-- Follow the
477
- [checklist](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/README.md#pull-request-checklist-for-python-plugins)
478
- to contribute the collector to the Netdata Agent [repository](https://github.com/netdata/netdata).
479
-- Check out the [example](https://github.com/netdata/netdata/tree/master/collectors/python.d.plugin/example) Python
480
- collector, which is a minimal example collector you could also use as a starting point. Once comfortable with that,
481
- then browse other [existing collectors](https://github.com/netdata/netdata/tree/master/collectors/python.d.plugin)
482
- that might have similarities to what you want to do.
483
-- If you're developing a proof of concept (PoC), consider migrating the collector in Golang
484
- ([go.d.plugin](https://github.com/netdata/go.d.plugin)) once you validate its value in production. Golang is more
485
- performant, easier to maintain, and simpler for users since it doesn't require a particular runtime on the node to
486
- execute (Python plugins require Python on the machine to be executed). Netdata uses Go as the platform of choice for
487
- production-grade collectors.
488
-- Celebrate! You have contributed to an open-source project with hundreds of thousands of users!
628
+Sockets are accessed in non-blocking mode with 15 second timeout.
629
630
+After every execution of `_get_raw_data` socket is closed, to prevent this module needs to set `_keep_alive` variable to `True` and implement custom `_check_raw_data` method.
631
632
+`_check_raw_data` should take raw data and return `True` if all data is received otherwise it should return `False`. Also it should do it in fast and efficient way.
exporting/graphite/README.md
+103
-5
@@ -13,15 +13,36 @@ You can use the Graphite connector for
13
the [exporting engine](https://github.com/netdata/netdata/blob/master/exporting/README.md) to archive your agent's
14
metrics to Graphite providers for long-term storage, further analysis, or correlation with data from other sources.
15
16
+## Prerequisites
17
+
18
+You have already [installed Netdata](https://github.com/netdata/netdata/edit/master/packaging/installer/README.md) and
19
+Graphite.
20
+
21
## Configuration
22
18
-To enable data exporting to a Graphite database, run `./edit-config exporting.conf` in the Netdata configuration
19
-directory and set the following options:
23
+Begin by using `edit-config` to open the `exporting.conf` file.
24
+
25
+```bash
26
+cd /etc/netdata # Replace this path with your Netdata config directory
27
+sudo ./edit-config exporting.conf
28
+```
29
+
30
+Enable the exporting engine by setting `enabled` to `yes` in the `[exporting:global]` section.
31
+
32
+```conf
33
+[exporting:global]
34
+ enabled = yes
35
+```
36
+
37
+Next, configure the connector. Find the `[graphite:my_graphite_instance]` example section and uncomment the line.
38
+Set the `destination` setting to `localhost:2003`. By default, the Docker image for Graphite listens on port `2003` for
39
+incoming metrics. If you installed Graphite a different way, you may need to change the port accordingly.
40
41
```conf
22
-[graphite:my_graphite_instance]
42
+[graphite:netdata]
43
enabled = yes
44
destination = localhost:2003
45
+ ...
46
```
47
48
Add `:http` or `:https` modifiers to the connector type if you need to use other than a plaintext protocol. For
@@ -33,7 +54,84 @@ example: `graphite:http:my_graphite_instance`,
54
password = my_password
55
```
56
36
-The Graphite connector is further configurable using additional settings. See
37
-the [exporting reference doc](https://github.com/netdata/netdata/blob/master/exporting/README.md#options) for details.
57
+The final result for a remote, secured host should be the following:
58
+
59
+```conf
60
+[graphite:https:netdata]
61
+ enabled = yes
62
+ username = my_username
63
+ password = my_password
64
+ destination = remote_host_url:2003
65
+ # data source = average
66
+ # prefix = netdata
67
+ # hostname = my_hostname
68
+ # update every = 10
69
+ # buffer on failures = 10
70
+ # timeout ms = 20000
71
+ # send names instead of ids = yes
72
+ # send charts matching = *
73
+ # send hosts matching = localhost *
74
+```
75
+
76
+We'll not worry about the [rest of the settings](https://github.com/netdata/netdata/blob/master/exporting/README.md#options)
77
+ for now. Restart the Agent using `sudo systemctl restart netdata`, or the
78
+[appropriate method](https://github.com/netdata/netdata/blob/master/docs/configure/start-stop-restart.md) for your
79
+system, to spin up the exporting engine.
80
+
81
+## See and organize Netdata metrics in Graphite
82
+
83
+Head back to the Graphite interface again, then click on the **Dashboard** link to get started with Netdata's exported
84
+metrics. You can also navigate directly to `http://NODE/dashboard`.
85
+
86
+Let's switch the interface to help you understand which metrics Netdata is exporting to Graphite. Click on **Dashboard**
87
+and **Configure UI**, then choose the **Tree** option. Refresh your browser to change the UI.
88
+
89
+
90
+
91
+You should now see a tree of available contexts, including one that matches the hostname of the Agent exporting metrics.
92
+In this example, the Agent's hostname is `arcturus`.
93
+
94
+Let's add some system CPU charts so you can monitor the long-term health of your system. Click through the tree to find
95
+**hostname → system → cpu** metrics, then click on the **user** context. A chart with metrics from that context appears
96
+in the dashboard. Add a few other system CPU charts to flesh things out.
97
+
98
+Next, let's combine one or two of these charts. Click and drag one chart onto the other, and wait until the green **Drop
99
+to merge** dialog appears. Release to merge the charts.
100
+
101
+
102
+
103
+Finally, save your dashboard. Click **Dashboard**, then **Save As**, then choose a name. Your dashboard is now saved.
104
+
105
+Of course, this is just the beginning of the customization you can do with Graphite. You can change the time range,
106
+share your dashboard with others, or use the composer to customize the size and appearance of specific charts. Learn
107
+more about adding, modifying, and combining graphs in
108
+the [Graphite docs](https://graphite.readthedocs.io/en/latest/dashboard.html).
109
+
110
+## Monitor the exporting engine
111
+
112
+As soon as the exporting engine begins, Netdata begins reporting metrics about the system's health and performance.
113
+
114
+
115
+
116
+You can use these charts to verify that Netdata is properly exporting metrics to Graphite. You can even add these
117
+exporting charts to your Graphite dashboard!
118
+
119
+### Add exporting charts to Netdata Cloud
120
+
121
+You can also show these exporting engine metrics on Netdata Cloud. If you don't have an account already,
122
+go [sign in](https://app.netdata.cloud) and get started for free.
123
+
124
+Add more metrics to a War Room's Nodes view by clicking on the **Add metric** button, then typing `exporting` into the
125
+context field. Choose the exporting contexts you want to add, then click **Add**. You'll see these charts alongside any
126
+others you've customized in Netdata Cloud.
127
+
128
+
129
+
130
+## What's next
131
132
+What you do with your exported metrics is entirely up to you, but as you might have seen in the Graphite connector
133
+configuration block, there are many other ways to tweak and customize which metrics you export to Graphite and how
134
+often.
135
136
+For full details about each configuration option and what it does, see
137
+the [exporting reference guide](https://github.com/netdata/netdata/blob/master/exporting/README.md).