apps.plugin optimization and new grouping of applications
Costa Tsaousis (ktsaou) committed
Jun 30, 2016 at 21:49 UTC
071b3bc550fc1dcdb708f6235a66b202645362c0
3 files changed
+199
-92
conf.d/apps_groups.conf
+157
-50
@@ -28,7 +28,7 @@
28
# *name* substring mode: will search for 'name' in the whole command line (/proc/PID/cmdline)
29
#
30
# If you enter even just one *name* (substring), apps.plugin will process
31
-# /proc/PID/cmdline for all processes, on every iteration.
31
+# /proc/PID/cmdline for all processes, just once (when they are first seen).
32
#
33
# To add process names with single quotes, enclose them in double quotes
34
# example: "process with this ' single quote"
@@ -44,61 +44,168 @@
44
# You can add any number of groups you like. Only the ones found running will
45
# affect the charts generated. However, producing charts with hundreds of
46
# dimensions may slow down your web browser.
47
+#
48
+# The order of the entries in this list is important: the first that matches
49
+# a process is used, so put important ones at the top. Processes not matched
50
+# by any row, will inherit it from their parents or children.
51
+#
52
+# The order also controls the order of the dimensions on the generated charts
53
+# (although applications started after apps.plugin is started, will be appended
54
+# to the existing list of dimensions the netdata daemon maintains).
55
+
56
+# -----------------------------------------------------------------------------
57
+# NETDATA processes accounting
58
+
59
+# netdata main process
60
+netdata: netdata
61
48
-compile: cc1 cc1plus as gcc* ld make automake autoconf git
49
-rsync: rsync
50
-media: mplayer vlc xine mediatomb omxplayer* kodi* xbmc* mediacenter eventlircd mpd minidlnad
51
-squid: squid* c-icap
52
-apache: apache* httpd
53
-mysql: mysql*
54
-asterisk: asterisk
55
-opensips: opensips* stund
56
-radius: radius*
62
+# netdata known plugins
63
+# plugins not defined here will be accumulated in netdata, above
64
+apps.plugin: apps.plugin
65
+charts.d.plugin: *charts.d.plugin*
66
+node.d.plugin: *node.d.plugin*
67
+python.d.plugin: *python.d.plugin*
68
+tc-qos-helper: *tc-qos-helper.sh*
69
+
70
+# -----------------------------------------------------------------------------
71
+# authentication/authorization related servers
72
+
73
+auth: radius* openldap* ldap*
74
fail2ban: fail2ban*
58
-mail: dovecot imapd pop3d
59
-postfix: master
60
-nginx: nginx
61
-splunk: splunkd
62
-mongo: mongod
63
-redis: redis*
64
-lighttpd: lighttpd
75
+
76
+# -----------------------------------------------------------------------------
77
+# web/ftp servers
78
+
79
+httpd: apache* httpd nginx* lighttpd
80
+proxy: squid* c-icap squidGuard varnish*
81
+php: php*
82
ftpd: proftpd in.tftpd
83
+
84
+# -----------------------------------------------------------------------------
85
+# database servers
86
+
87
+sql: mysqld* mariad* postgres*
88
+nosql: mongod redis*
89
+
90
+# -----------------------------------------------------------------------------
91
+# email servers
92
+
93
+email: dovecot imapd pop3d amavis* master zmstat* zmmailboxdmgr
94
+
95
+# -----------------------------------------------------------------------------
96
+# networking and VPN servers
97
+
98
+ppp: ppp*
99
+vpn: openvpn pptp* cjdroute
100
+wifi: hostapd wpa_supplicant
101
+
102
+# -----------------------------------------------------------------------------
103
+# high availability and balancers
104
+
105
+balancer: ipvs_* haproxy
106
+ha: corosync hs_logd ha_logd stonithd
107
+
108
+# -----------------------------------------------------------------------------
109
+# telephony
110
+
111
+pbx: asterisk safe_asterisk *vicidial*
112
+sip: opensips* stund
113
+
114
+# -----------------------------------------------------------------------------
115
+# monitoring
116
+
117
+logs: ulogd* syslog* rsyslog* logrotate
118
+nms: snmpd vnstatd smokeping zabbix* monit munin* mon openhpid watchdog
119
+splunk: splunkd
120
+
121
+# -----------------------------------------------------------------------------
122
+# file systems and file servers
123
+
124
samba: smbd nmbd winbindd
125
nfs: rpcbind rpc.* nfs*
126
+zfs: spl_* z_* txg_* zil_* arc_* l2arc*
127
+btrfs: btrfs*
128
+
129
+# -----------------------------------------------------------------------------
130
+# containers & virtual machines
131
+
132
+containers: lxc* docker*
133
+VMs: vbox* VBox* qemu*
134
+
135
+# -----------------------------------------------------------------------------
136
+# ssh servers and clients
137
+
138
ssh: ssh* scp
69
-X: X lightdm xdm pulseaudio gkrellm
70
-xfce: xfwm4 xfdesktop xfce* Thunar xfsettingsd
71
-gnome: gnome-* gdm gconfd-2
72
-named: named rncd
73
-clam: clam* *clam
74
-cups: cups*
75
-ntp: ntp*
76
-torrent: deluge* transmission*
77
-vbox: vbox* VBox*
78
-log: ulogd syslog* rsyslog* logrotate
79
-nms: snmpd vnstatd smokeping zabbix* monit munin* mon openhpid
80
-ppp: ppp* pptp*
81
-inetd: inetd xinetd
82
-openvpn: openvpn
83
-cjdns: cjdroute
84
-cron: cron atd
85
-ha: corosync hs_logd ha_logd stonithd
86
-ipvs: ipvs_*
139
+
140
+# -----------------------------------------------------------------------------
141
+# print servers and clients
142
+
143
+print: cups* lpd lpq
144
+
145
+# -----------------------------------------------------------------------------
146
+# time servers and clients
147
+
148
+time: ntp*
149
+
150
+# -----------------------------------------------------------------------------
151
+# dhcp servers and clients
152
+
153
+dhcp: *dhcp*
154
+
155
+# -----------------------------------------------------------------------------
156
+# name servers and clients
157
+
158
+named: named rncd dig
159
+
160
+# -----------------------------------------------------------------------------
161
+# installation / compilation / debugging
162
+
163
+build: cc1 cc1plus as gcc* ld make automake autoconf autoreconf git valgrind*
164
+
165
+# -----------------------------------------------------------------------------
166
+# antivirus
167
+
168
+antivirus: clam* *clam
169
+
170
+# -----------------------------------------------------------------------------
171
+# torrent clients
172
+
173
+torrents: *deluge* transmission* *SickBeard*
174
+
175
+# -----------------------------------------------------------------------------
176
+# backup servers and clients
177
+
178
+backup: rsync bacula*
179
+
180
+# -----------------------------------------------------------------------------
181
+# cron
182
+
183
+cron: cron atd anacron
184
+
185
+# -----------------------------------------------------------------------------
186
+# UPS
187
+
188
+ups: upsmon upsd */nut/*
189
+
190
+# -----------------------------------------------------------------------------
191
+# Kernel / System
192
+
193
+system: systemd* udisks* udevd* *udevd connmand ipv6_addrconf dbus-* inetd xinetd mdadm
194
kernel: kthreadd kauditd lockd khelper kdevtmpfs khungtaskd rpciod fsnotify_mark kthrotld iscsi_eh deferwq
88
-netdata: netdata
89
-crsproxy: crsproxy
90
-wifi: hostapd wpa_supplicant
91
-system: systemd* udisks* udevd connmand ipv6_addrconf dbus-*
195
ksmd: ksmd
93
-lxc: lxc*
94
-zfs-spl: spl_*
95
-zfs-posix: z_*
96
-zfs-txg: txg_* zil_*
97
-zfs-arc: arc_* l2arc*
98
-php: php*
99
-zimbra: zmstat* zmmailboxdmgr
196
+
197
+# -----------------------------------------------------------------------------
198
+# media players, servers, clients
199
+
200
+media: mplayer vlc xine mediatomb omxplayer* kodi* xbmc* mediacenter eventlircd mpd minidlnad mt-daapd avahi*
201
+
202
+# -----------------------------------------------------------------------------
203
+# X
204
+
205
+X: X lightdm xdm pulseaudio gkrellm xfwm4 xfdesktop xfce* Thunar xfsettingsd xfconfd gnome-* gdm gconfd-2 *gvfsd gvfsd* kdm slim
206
+
207
+# -----------------------------------------------------------------------------
208
+# other application servers
209
+
210
+crsproxy: crsproxy
211
java: java
101
-bacula: bacula*
102
-amavis: amavis*
103
-varnish: varnish*
104
-haproxy: haproxy
src/apps_plugin.c
+39
-39
@@ -1,7 +1,3 @@
1
-// TODO
2
-//
3
-// 1. disable RESET_OR_OVERFLOW check in charts
4
-
1
#ifdef HAVE_CONFIG_H
2
#include <config.h>
3
#endif
@@ -309,10 +305,12 @@ struct target *get_apps_groups_target(const char *id, struct target *target)
305
}
306
uint32_t hash = simple_hash(id);
307
312
- struct target *w;
308
+ struct target *w, *last = apps_groups_root_target;
309
for(w = apps_groups_root_target ; w ; w = w->next) {
310
if(w->idhash == hash && strncmp(nid, w->id, MAX_NAME) == 0)
311
return w;
312
+
313
+ last = w;
314
}
315
316
w = calloc(sizeof(struct target), 1);
@@ -344,8 +342,9 @@ struct target *get_apps_groups_target(const char *id, struct target *target)
342
w->debug = tdebug;
343
w->target = target;
344
347
- w->next = apps_groups_root_target;
348
- apps_groups_root_target = w;
345
+ // append it, to maintain the order in apps_groups.conf
346
+ if(last) last->next = w;
347
+ else apps_groups_root_target = w;
348
349
if(unlikely(debug))
350
fprintf(stderr, "apps.plugin: ADDING TARGET ID '%s', process name '%s' (%s), aggregated on target '%s', options: %s %s\n"
@@ -1152,32 +1151,34 @@ int collect_data_for_all_processes_from_proc(void)
1151
all_pids_count = 0;
1152
for(p = root_of_pids; p ; p = p->next) {
1153
all_pids_count++;
1155
- p->parent = NULL;
1156
- p->updated = 0;
1157
- p->children_count = 0;
1158
- p->merged = 0;
1159
- p->new_entry = 0;
1160
-
1161
- p->last_minflt = p->minflt;
1162
- p->last_majflt = p->majflt;
1163
- p->last_utime = p->utime;
1164
- p->last_stime = p->stime;
1165
-
1166
- p->last_cminflt = p->cminflt;
1167
- p->last_cmajflt = p->cmajflt;
1168
- p->last_cutime = p->cutime;
1169
- p->last_cstime = p->cstime;
1154
+
1155
+ p->parent = NULL;
1156
+
1157
+ p->updated = 0;
1158
+ p->children_count = 0;
1159
+ p->merged = 0;
1160
+ p->new_entry = 0;
1161
+
1162
+ p->last_minflt = p->minflt;
1163
+ p->last_majflt = p->majflt;
1164
+ p->last_utime = p->utime;
1165
+ p->last_stime = p->stime;
1166
+
1167
+ p->last_cminflt = p->cminflt;
1168
+ p->last_cmajflt = p->cmajflt;
1169
+ p->last_cutime = p->cutime;
1170
+ p->last_cstime = p->cstime;
1171
1172
p->last_fix_cminflt = p->fix_cminflt;
1173
p->last_fix_cmajflt = p->fix_cmajflt;
1174
p->last_fix_cutime = p->fix_cutime;
1175
p->last_fix_cstime = p->fix_cstime;
1176
1176
- p->last_io_logical_bytes_read = p->io_logical_bytes_read;
1177
+ p->last_io_logical_bytes_read = p->io_logical_bytes_read;
1178
p->last_io_logical_bytes_written = p->io_logical_bytes_written;
1178
- p->last_io_read_calls = p->io_read_calls;
1179
- p->last_io_write_calls = p->io_write_calls;
1180
- p->last_io_storage_bytes_read = p->io_storage_bytes_read;
1179
+ p->last_io_read_calls = p->io_read_calls;
1180
+ p->last_io_write_calls = p->io_write_calls;
1181
+ p->last_io_storage_bytes_read = p->io_storage_bytes_read;
1182
p->last_io_storage_bytes_written = p->io_storage_bytes_written;
1183
p->last_io_cancelled_write_bytes = p->io_cancelled_write_bytes;
1184
}
@@ -1214,21 +1215,11 @@ int collect_data_for_all_processes_from_proc(void)
1215
p->ppid = 0;
1216
}
1217
1217
- // --------------------------------------------------------------------
1218
- // /proc/<pid>/cmdline
1219
-
1220
- if(proc_pid_cmdline_is_needed) {
1221
- if(unlikely(read_proc_pid_cmdline(p))) {
1222
- error("Cannot process %s/proc/%d/cmdline", host_prefix, pid);
1223
- }
1224
- }
1225
-
1218
// --------------------------------------------------------------------
1219
// /proc/<pid>/statm
1220
1221
if(unlikely(read_proc_pid_statm(p))) {
1230
- error("Cannot process %s/proc/%d/statm", host_prefix, pid);
1231
-
1222
+ error("Cannot process %s/proc/%d/statm", host_prefix, pid);
1223
// there is no reason to proceed if we cannot get its memory status
1224
continue;
1225
}
@@ -1258,11 +1249,18 @@ int collect_data_for_all_processes_from_proc(void)
1249
// check if it is target
1250
// we do this only once, the first time this pid is loaded
1251
if(unlikely(p->new_entry)) {
1252
+ // /proc/<pid>/cmdline
1253
+ if(proc_pid_cmdline_is_needed) {
1254
+ if(unlikely(read_proc_pid_cmdline(p))) {
1255
+ error("Cannot process %s/proc/%d/cmdline", host_prefix, pid);
1256
+ }
1257
+ }
1258
+
1259
if(unlikely(debug))
1262
- fprintf(stderr, "apps.plugin: \tJust added %s\n", p->comm);
1260
+ fprintf(stderr, "apps.plugin: \tJust added %d (%s)\n", pid, p->comm);
1261
1262
uint32_t hash = simple_hash(p->comm);
1265
- size_t pclen = strlen(p->comm);
1263
+ size_t pclen = strlen(p->comm);
1264
1265
struct target *w;
1266
for(w = apps_groups_root_target; w ; w = w->next) {
@@ -1283,6 +1281,8 @@ int collect_data_for_all_processes_from_proc(void)
1281
1282
if(debug || (p->target && p->target->debug))
1283
fprintf(stderr, "apps.plugin: \t\t%s linked to target %s\n", p->comm, p->target->name);
1284
+
1285
+ break;
1286
}
1287
}
1288
}
web/index.html
+3
-3
@@ -1326,19 +1326,19 @@ var menuData = {
1326
1327
'apps': {
1328
title: 'Applications',
1329
- info: 'Per application statistics are collected using netdata\'s <code>apps.plugin</code>. This plugin walks through the entire <code>/proc</code> filesystem and aggregates statistics for applications of interest, defined in <code>/etc/netdata/apps_groups.conf</code> (the default is <a href="https://github.com/firehol/netdata/blob/master/conf.d/apps_groups.conf" target="_blank">here</a>). The plugin internally builds a process tree (much like <code>ps fax</code> does), and groups processes together (evaluating both child and parent processes) so that the result is always a chart with a predefined set of dimensions (of course, only application groups found running are reported).',
1329
+ info: 'Per application statistics are collected using netdata\'s <code>apps.plugin</code>. This plugin walks through the entire <code>/proc</code> filesystem and aggregates statistics for applications of interest, defined in <code>/etc/netdata/apps_groups.conf</code> (the default is <a href="https://github.com/firehol/netdata/blob/master/conf.d/apps_groups.conf" target="_blank">here</a>). The plugin internally builds a process tree (much like <code>ps fax</code> does), and groups processes together (evaluating both child and parent processes) so that the result is always a chart with a predefined set of dimensions (of course, only application groups found running are reported). The reported values are compatible with <code>top</code>, although the netdata plugin counts also the resources of exited children (unlike <code>top</code> which shows only the resources of the currently running processes). So for processes like shell scripts, the reported values include the resources used by the commands these scripts run within each timeframe.',
1330
height: 1.5
1331
},
1332
1333
'users': {
1334
title: 'Users',
1335
- info: 'Per user statistics are collected using netdata\'s <code>apps.plugin</code>. This plugin walks through the entire <code>/proc</code> filesystem and aggregates statistics per user.',
1335
+ info: 'Per user statistics are collected using netdata\'s <code>apps.plugin</code>. This plugin walks through the entire <code>/proc</code> filesystem and aggregates statistics per user. The reported values are compatible with <code>top</code>, although the netdata plugin counts also the resources of exited children (unlike <code>top</code> which shows only the resources of the currently running processes). So for processes like shell scripts, the reported values include the resources used by the commands these scripts run within each timeframe.',
1336
height: 1.5
1337
},
1338
1339
'groups': {
1340
title: 'User Groups',
1341
- info: 'Per user group statistics are collected using netdata\'s <code>apps.plugin</code>. This plugin walks through the entire <code>/proc</code> filesystem and aggregates statistics per user group.',
1341
+ info: 'Per user group statistics are collected using netdata\'s <code>apps.plugin</code>. This plugin walks through the entire <code>/proc</code> filesystem and aggregates statistics per user group. The reported values are compatible with <code>top</code>, although the netdata plugin counts also the resources of exited children (unlike <code>top</code> which shows only the resources of the currently running processes). So for processes like shell scripts, the reported values include the resources used by the commands these scripts run within each timeframe.',
1342
height: 1.5
1343
},
1344