@cryptotaxi247 / netdata-1
Code Commits Issues Pulls Sessions CI/CD
master
master
yaml 307 lines 14.9 KB
Raw
1 plugin_name: python.d.plugin
2 modules:
3 - meta:
4 plugin_name: python.d.plugin
5 module_name: pandas
6 monitored_instance:
7 name: Pandas
8 link: https://pandas.pydata.org/
9 categories:
10 - data-collection.databases
11 icon_filename: pandas.png
12 related_resources:
13 integrations:
14 list: []
15 info_provided_to_referring_integrations:
16 description: ""
17 keywords:
18 - pandas
19 - python
20 overview:
21 data_collection:
22 metrics_description: |
23 [Pandas](https://pandas.pydata.org/) is a de-facto standard in reading and processing most types of structured data in Python.
24 If you have metrics appearing in a CSV, JSON, XML, HTML, or [other supported format](https://pandas.pydata.org/docs/user_guide/io.html),
25 either locally or via some HTTP endpoint, you can easily ingest and present those metrics in Netdata, by leveraging the Pandas collector.
26
27 This collector can be used to collect pretty much anything that can be read by Pandas, and then processed by Pandas.
28 method_description: |
29 The collector uses [pandas](https://pandas.pydata.org/) to pull data and do pandas-based preprocessing, before feeding to Netdata.
30 supported_platforms:
31 include: []
32 exclude: []
33 multi_instance: true
34 additional_permissions:
35 description: ""
36 default_behavior:
37 auto_detection:
38 description: ""
39 limits:
40 description: ""
41 performance_impact:
42 description: ""
43 setup:
44 prerequisites:
45 list:
46 - title: Python Requirements
47 description: |
48 This collector depends on some Python (Python 3 only) packages that can usually be installed via `pip` or `pip3`.
49
50 ```bash
51 sudo pip install pandas requests
52 ```
53
54 Note: If you would like to use [`pandas.read_sql`](https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html) to query a database, you will need to install the below packages as well.
55
56 ```bash
57 sudo pip install 'sqlalchemy<2.0' psycopg2-binary
58 ```
59 configuration:
60 file:
61 name: python.d/pandas.conf
62 description: ""
63 options:
64 description: |
65 There are 2 sections:
66
67 * Global variables
68 * One or more JOBS that can define multiple different instances to monitor.
69
70 The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values.
71
72 Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition.
73
74 Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified.
75 folding:
76 title: Config options
77 enabled: true
78 list:
79 - name: chart_configs
80 description: an array of chart configuration dictionaries
81 default_value: "[]"
82 required: true
83 - name: chart_configs.name
84 description: name of the chart to be displayed in the dashboard.
85 default_value: None
86 required: true
87 - name: chart_configs.title
88 description: title of the chart to be displayed in the dashboard.
89 default_value: None
90 required: true
91 - name: chart_configs.family
92 description: "[family](/docs/dashboards-and-charts/netdata-charts.md#families) of the chart to be displayed in the dashboard."
93 default_value: None
94 required: true
95 - name: chart_configs.context
96 description: "[context](/docs/dashboards-and-charts/netdata-charts.md#contexts) of the chart to be displayed in the dashboard."
97 default_value: None
98 required: true
99 - name: chart_configs.type
100 description: the type of the chart to be displayed in the dashboard.
101 default_value: None
102 required: true
103 - name: chart_configs.units
104 description: the units of the chart to be displayed in the dashboard.
105 default_value: None
106 required: true
107 - name: chart_configs.df_steps
108 description: a series of pandas operations (one per line) that each returns a dataframe.
109 default_value: None
110 required: true
111 - name: update_every
112 description: Sets the default data collection frequency.
113 default_value: 5
114 required: false
115 - name: priority
116 description: Controls the order of charts at the netdata dashboard.
117 default_value: 60000
118 required: false
119 - name: autodetection_retry
120 description: Sets the job re-check interval in seconds.
121 default_value: 0
122 required: false
123 - name: penalty
124 description: Indicates whether to apply penalty to update_every in case of failures.
125 default_value: yes
126 required: false
127 - name: name
128 description: Job name. This value will overwrite the `job_name` value. JOBS with the same name are mutually exclusive. Only one of them will be allowed running at any time. This allows autodetection to try several alternatives and pick the one that works.
129 default_value: ""
130 required: false
131 examples:
132 folding:
133 enabled: true
134 title: Config
135 list:
136 - name: Temperature API Example
137 folding:
138 enabled: true
139 description: example pulling some hourly temperature data, a chart for today forecast (mean,min,max) and another chart for current.
140 config: |
141 temperature:
142 name: "temperature"
143 update_every: 5
144 chart_configs:
145 - name: "temperature_forecast_by_city"
146 title: "Temperature By City - Today Forecast"
147 family: "temperature.today"
148 context: "pandas.temperature"
149 type: "line"
150 units: "Celsius"
151 df_steps: >
152 pd.DataFrame.from_dict(
153 {city: requests.get(f'https://api.open-meteo.com/v1/forecast?latitude={lat}&longitude={lng}&hourly=temperature_2m').json()['hourly']['temperature_2m']
154 for (city,lat,lng)
155 in [
156 ('dublin', 53.3441, -6.2675),
157 ('athens', 37.9792, 23.7166),
158 ('london', 51.5002, -0.1262),
159 ('berlin', 52.5235, 13.4115),
160 ('paris', 48.8567, 2.3510),
161 ('madrid', 40.4167, -3.7033),
162 ('new_york', 40.71, -74.01),
163 ('los_angeles', 34.05, -118.24),
164 ]
165 }
166 );
167 df.describe(); # get aggregate stats for each city;
168 df.transpose()[['mean', 'max', 'min']].reset_index(); # just take mean, min, max;
169 df.rename(columns={'index':'city'}); # some column renaming;
170 df.pivot(columns='city').mean().to_frame().reset_index(); # force to be one row per city;
171 df.rename(columns={0:'degrees'}); # some column renaming;
172 pd.concat([df, df['city']+'_'+df['level_0']], axis=1); # add new column combining city and summary measurement label;
173 df.rename(columns={0:'measurement'}); # some column renaming;
174 df[['measurement', 'degrees']].set_index('measurement'); # just take two columns we want;
175 df.sort_index(); # sort by city name;
176 df.transpose(); # transpose so its just one wide row;
177 - name: "temperature_current_by_city"
178 title: "Temperature By City - Current"
179 family: "temperature.current"
180 context: "pandas.temperature"
181 type: "line"
182 units: "Celsius"
183 df_steps: >
184 pd.DataFrame.from_dict(
185 {city: requests.get(f'https://api.open-meteo.com/v1/forecast?latitude={lat}&longitude={lng}&current_weather=true').json()['current_weather']
186 for (city,lat,lng)
187 in [
188 ('dublin', 53.3441, -6.2675),
189 ('athens', 37.9792, 23.7166),
190 ('london', 51.5002, -0.1262),
191 ('berlin', 52.5235, 13.4115),
192 ('paris', 48.8567, 2.3510),
193 ('madrid', 40.4167, -3.7033),
194 ('new_york', 40.71, -74.01),
195 ('los_angeles', 34.05, -118.24),
196 ]
197 }
198 );
199 df.transpose();
200 df[['temperature']];
201 df.transpose();
202 - name: API CSV Example
203 folding:
204 enabled: true
205 description: example showing a read_csv from a url and some light pandas data wrangling.
206 config: |
207 example_csv:
208 name: "example_csv"
209 update_every: 2
210 chart_configs:
211 - name: "london_system_cpu"
212 title: "London System CPU - Ratios"
213 family: "london_system_cpu"
214 context: "pandas"
215 type: "line"
216 units: "n"
217 df_steps: >
218 pd.read_csv('https://london.my-netdata.io/api/v1/data?chart=system.cpu&format=csv&after=-60', storage_options={'User-Agent': 'netdata'});
219 df.drop('time', axis=1);
220 df.mean().to_frame().transpose();
221 df.apply(lambda row: (row.user / row.system), axis = 1).to_frame();
222 df.rename(columns={0:'average_user_system_ratio'});
223 df*100;
224 - name: API JSON Example
225 folding:
226 enabled: true
227 description: example showing a read_json from a url and some light pandas data wrangling.
228 config: |
229 example_json:
230 name: "example_json"
231 update_every: 2
232 chart_configs:
233 - name: "london_system_net"
234 title: "London System Net - Total Bandwidth"
235 family: "london_system_net"
236 context: "pandas"
237 type: "area"
238 units: "kilobits/s"
239 df_steps: >
240 pd.DataFrame(requests.get('https://london.my-netdata.io/api/v1/data?chart=system.net&format=json&after=-1').json()['data'], columns=requests.get('https://london.my-netdata.io/api/v1/data?chart=system.net&format=json&after=-1').json()['labels']);
241 df.drop('time', axis=1);
242 abs(df);
243 df.sum(axis=1).to_frame();
244 df.rename(columns={0:'total_bandwidth'});
245 - name: XML Example
246 folding:
247 enabled: true
248 description: example showing a read_xml from a url and some light pandas data wrangling.
249 config: |
250 example_xml:
251 name: "example_xml"
252 update_every: 2
253 line_sep: "|"
254 chart_configs:
255 - name: "temperature_forcast"
256 title: "Temperature Forecast"
257 family: "temp"
258 context: "pandas.temp"
259 type: "line"
260 units: "celsius"
261 df_steps: >
262 pd.read_xml('http://metwdb-openaccess.ichec.ie/metno-wdb2ts/locationforecast?lat=54.7210798611;long=-8.7237392806', xpath='./product/time[1]/location/temperature', parser='etree')|
263 df.rename(columns={'value': 'dublin'})|
264 df[['dublin']]|
265 - name: SQL Example
266 folding:
267 enabled: true
268 description: example showing a read_sql from a postgres database using sqlalchemy.
269 config: |
270 sql:
271 name: "sql"
272 update_every: 5
273 chart_configs:
274 - name: "sql"
275 title: "SQL Example"
276 family: "sql.example"
277 context: "example"
278 type: "line"
279 units: "percent"
280 df_steps: >
281 pd.read_sql_query(
282 sql='\
283 select \
284 random()*100 as metric_1, \
285 random()*100 as metric_2 \
286 ',
287 con=create_engine('postgresql://localhost/postgres?user=netdata&password=netdata')
288 );
289 troubleshooting:
290 problems:
291 list: []
292 alerts: []
293 metrics:
294 folding:
295 title: Metrics
296 enabled: false
297 description: |
298 This collector is expecting one row in the final pandas DataFrame. It is that first row that will be taken
299 as the most recent values for each dimension on each chart using (`df.to_dict(orient='records')[0]`).
300 See [pd.to_dict()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_dict.html)."
301 availability: []
302 scopes:
303 - name: global
304 description: |
305 These metrics refer to the entire monitored application.
306 labels: []
307 metrics: []