@cryptotaxi247 / netdata-1 / commits / 696264006

eBPF process plugin (#7979)

* syscall_plugin: Compilation This commit brings the necessaries changes to the compilation files * syscall_plugin: Collector body This commit brings the collector body to files. * syscall_plugin: .gitignore This commit adds syscall.plugin to .gitignore * syscall_plugin: Plugin adjust Fix reference and remove message * syscall_plugin: Remove limit Remove call to setrlimit * syscall: Fix start This commit fixes problems related with start of the plugin * syscall_plugin: Bring heartbeat This commit removes the sleep and changes to heartbeat to avoid plugin receive a SIGTERM * syscall_plugin: Missing semicolon * syscall_plugin: Fix dimension Brings the initial value of chart for the normal dimension of the other values * syscall_plugin: Fix dimension 2 The previous change did not give the expected results, so I am bringing more a fix * syscall_plugin: adjust values Rename function and adjust pid size * syscall_plugin: Remove Chart and fix var this commit removes a chart that will not be created and fix an error when the bytes were calculated * syscall_plugin: Brings error This commit brings a new variable that will be used to identify errors * syscall_plugin: Rename charts This commit starts to rename the charts properly * syscall_plugin: Rename plugin * syscall_plugin: missing changes for rename * syscall_plugin: fix compilation * syscall_plugin: bring new charts * syscall_plugin: Warnings Remove warnings from compilation time * vfs_plugin: Fix Error chart plot There was an error when the chart was being displayed * vfs_plugin: Change family This commit changes the family of the VFS plugin * vfs_plugin: Fix order This PR fixes the wrong order when creating a chart * vfs_plugin: Remove path Remove path from structure * vfs_plugin: From Perf to HASH This commit converts the main source a hash table and also split the data collection per chart * vfs_plugin: Adjusts and exit This commit brings adjusts to the collect and the complete monitor to exit events * vfs_plugin: Start process This commit brings the monitoring of a process start and thread creation to Netdata * vfs_plugin: Visualization and collection Adjust variables to show and to collect data * vfs_plugin: Connection with apps plugin This commit starts to bring the connection with apps. * vfs_plugin: Various This commit brings new label for charts, fix to error chart and adjusts for new charts, I am sorry * vfs_plugin: basis new chart This commit brings the basis of the new charts for the plugin * vfs_plugin: Apps plugin This commit brings the integration with apps.plugin * vfs_plugin:fix counter This commit fixer the difference between apps plugin and counter * ebpf_plugin: rename charts This commit renames the charts * ebpf_plugin: New charts adjusts and log start * ebpf_plugin: Log thread Creates the log thread that will be used to store error message * ebpf_plugin: Rename Web Group This commit reorganize the charts on dashboard * ebpf_plugin: Restore This commit restore the previous status of the collector where we only have a global vision of the problems * ebpf_plugin: kretprobe This commit brings the initial changes for the collector works with both eBPF program * ebpf_plugin: New syscalls This commit brings the new syscalls that we are monitoring * ebpf_plugin: New charts This commit brings new charts to the collector * ebpf_plugin: Parse config This commit starts the parser of the file * ebpf_plugin: collector debug * ebpf_plugin: Global variables from config This commit brings the global variable update from the config file * ebpf_plugin: Clean kprobe_events This commit brings the clean of kprobe_events and also starts the common library for all eBPF collectors * ebpf_plugin: Check kernel version This function brings a check for the kernel version * ebpf_plugin: Start documentation This commit brings the initial documentation for the users * ebpf_plugin: Documentation This commit brings adjust to code and updates for the documentation * ebpf_plugin: this commit brings the developer mode to the collector * ebpf_plugin: Documentation This commit brings more information to the documentation * ebpf_plugin: Documentation This commit brings more information to the documentation * ebpf_plugin: errno to logs Brings errno number to logs * ebpf_plugin: Documentation This commit brings fixes to the collector documentation * ebpf_plugin: Move description This commit move the chart description from the C code to dashboard_info.js * ebpf_plugin: Rename files This commit rename files to the final version * ebpf_plugin: COntinue renaming This commit continue renaming the files to the final version * ebpf_plugin: Renaming process This commit renames the final plugin * ebpf_plugin: Finish rename This commit finishes the rename processing * ebpf_plugin: fix entry charts This commit removes one chart from mode * ebpf_plugin: Fix remove This commit brings a new function to fix the unload of collector when the collector is running in entry mode * ebpf_plugin: Rename on old kernels This commit brings fixes for syscall names * ebpf_plugin: Timestamp to log This commit brings the timestamp to the logs * ebpf_plugin: Remove syscall With the changes on the backend, we are not monitoring more sys_clone * ebpf_plugin: The syscall is important for 5.3 or newer, so I am returning * ebpf_plugin: Remove concurrency This commit adds variables necessary to interact with the new structor of the eBPF program * ebpf_plugin: Ids to dimension This commit fews the functions name as ids for the dimensions * ebpf_plugin: Missing chart This commit brings the missing chart for Netdata * ebpf_plugin: Remove unecessary message Remove unecessary error message from the collector * ebpf_plugin: Rename dimension This commit renames the dimension for something more meaninful * ebpf_plugin: Optional log This commit converts the developer.log in an optional feature * redirect to stdoou This commit starts to bring the capability to redirect everything to stdout * ebpf_plugin: Disable dev mode This commit removes the possibility to load the dev mode file for while * ebpf_plugin: Disable eBPF process By default this plugin won't be enabled * ebpf_plugin: Update debug message * ebpf_plugin: this commit adjusts documentation to next release. * ebpf_plugin: documentation fix. * ebpf_plugin: Percpu hash This commit moves from an unique hash table for various to speed up the collector * ebpf_plugin: Compatibility This commit set compatibility version between kernels

thiagoftsm committed Feb 17, 2020 at 21:28 UTC 696264006c1aa4ef111aab2be28f591340575c9e
20 files changed +1484 -2
.gitignore
+3
@@ -66,6 +66,9 @@ slabinfo.plugin
66 cgroup-network
67 !cgroup-network/
68
69 +ebpf_process.plugin
70 +!ebpf_process.plugin/
71 +
72 # protoc generated files
73 *.pb.cc
74 *.pb.h
CMakeLists.txt
+23
@@ -298,6 +298,8 @@ set(LIBNETDATA_FILES
298 libnetdata/clocks/clocks.h
299 libnetdata/dictionary/dictionary.c
300 libnetdata/dictionary/dictionary.h
301 + libnetdata/ebpf/ebpf.c
302 + libnetdata/ebpf/ebpf.h
303 libnetdata/eval/eval.c
304 libnetdata/eval/eval.h
305 libnetdata/inlined.h
@@ -405,6 +407,10 @@ set(SLABINFO_PLUGIN_FILES
407 collectors/slabinfo.plugin/slabinfo.c
408 )
409
410 +set(EBPF_PROCESS_PLUGIN_FILES
411 + collectors/ebpf_process.plugin/ebpf_process.c
412 + )
413 +
414 set(PROC_PLUGIN_FILES
415 collectors/proc.plugin/ipc.c
416 collectors/proc.plugin/plugin_proc.c
@@ -778,6 +784,7 @@ IF(LINUX)
784 SET(ENABLE_PLUGIN_APPS True)
785 SET(ENABLE_PLUGIN_PERF True)
786 SET(ENABLE_PLUGIN_SLABINFO True)
787 + SET(ENABLE_PLUGIN_EBPF True)
788
789 ELSEIF(FREEBSD)
790 add_executable(netdata config.h ${NETDATA_FILES} ${FREEBSD_PLUGIN_FILES})
@@ -788,6 +795,7 @@ ELSEIF(FREEBSD)
795 SET(ENABLE_PLUGIN_APPS True)
796 SET(ENABLE_PLUGIN_PERF False)
797 SET(ENABLE_PLUGIN_SLABINFO False)
798 + SET(ENABLE_PLUGIN_EBPF False)
799
800 ELSEIF(MACOS)
801 add_executable(netdata config.h ${NETDATA_FILES} ${MACOS_PLUGIN_FILES})
@@ -798,6 +806,7 @@ ELSEIF(MACOS)
806 SET(ENABLE_PLUGIN_APPS False)
807 SET(ENABLE_PLUGIN_PERF False)
808 SET(ENABLE_PLUGIN_SLABINFO False)
809 + SET(ENABLE_PLUGIN_EBPF False)
810
811 ENDIF()
812
@@ -904,6 +913,20 @@ ELSE()
913 ENDIF()
914
915
916 +# -----------------------------------------------------------------------------
917 +# ebpf_process.plugin
918 +
919 +IF(ENABLE_PLUGIN_EBPF)
920 + message(STATUS "ebpf_process.plugin: enabled")
921 + add_executable(ebpf_process.plugin config.h ${EBPF_PROCESS_PLUGIN_FILES})
922 + target_link_libraries (ebpf_process.plugin libnetdata ${NETDATA_COMMON_LIBRARIES})
923 + target_include_directories(ebpf_process.plugin PUBLIC ${NETDATA_COMMON_INCLUDE_DIRS})
924 + target_compile_options(ebpf_process.plugin PUBLIC ${NETDATA_COMMON_CFLAGS})
925 +ELSE()
926 + message(STATUS "ebpf_process.plugin: disabled")
927 +ENDIF()
928 +
929 +
930 # -----------------------------------------------------------------------------
931 # slabinfo.plugin
932
Makefile.am
+17
@@ -135,6 +135,8 @@ LIBNETDATA_FILES = \
135 libnetdata/clocks/clocks.h \
136 libnetdata/dictionary/dictionary.c \
137 libnetdata/dictionary/dictionary.h \
138 + libnetdata/ebpf/ebpf.c \
139 + libnetdata/ebpf/ebpf.h \
140 libnetdata/eval/eval.c \
141 libnetdata/eval/eval.h \
142 libnetdata/inlined.h \
@@ -255,6 +257,12 @@ PERF_PLUGIN_FILES = \
257 $(LIBNETDATA_FILES) \
258 $(NULL)
259
260 +EBPF_PROCESS_PLUGIN_FILES = \
261 + collectors/ebpf_process.plugin/ebpf_process.c \
262 + collectors/ebpf_process.plugin/ebpf_process.h \
263 + $(LIBNETDATA_FILES) \
264 + $(NULL)
265 +
266 PROC_PLUGIN_FILES = \
267 collectors/proc.plugin/ipc.c \
268 collectors/proc.plugin/plugin_proc.c \
@@ -585,6 +593,7 @@ NETDATA_COMMON_LIBS = \
593 $(OPTIONAL_JUDY_LIBS) \
594 $(OPTIONAL_SSL_LIBS) \
595 $(OPTIONAL_JSONC_LIBS) \
596 + $(OPTIONAL_EBPF_LIBS) \
597 $(NULL)
598
599 NETDATACLI_FILES = \
@@ -651,6 +660,14 @@ if ENABLE_PLUGIN_FREEIPMI
660 $(NULL)
661 endif
662
663 +if ENABLE_PLUGIN_EBPF
664 + plugins_PROGRAMS += ebpf_process.plugin
665 + ebpf_process_plugin_SOURCES = $(EBPF_PROCESS_PLUGIN_FILES)
666 + ebpf_process_plugin_LDADD = \
667 + $(NETDATA_COMMON_LIBS) \
668 + $(NULL)
669 +endif
670 +
671 if ENABLE_PLUGIN_CUPS
672 plugins_PROGRAMS += cups.plugin
673 cups_plugin_SOURCES = $(CUPS_PLUGIN_FILES)
collectors/Makefile.am
+1
@@ -24,6 +24,7 @@ SUBDIRS = \
24 python.d.plugin \
25 slabinfo.plugin \
26 statsd.plugin \
27 + ebpf_process.plugin \
28 tc.plugin \
29 $(NULL)
30
collectors/ebpf_process.plugin/Makefile.am new
+12
@@ -0,0 +1,12 @@
1 +# SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +AUTOMAKE_OPTIONS = subdir-objects
4 +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in
5 +
6 +dist_noinst_DATA = \
7 + README.md \
8 + $(NULL)
9 +
10 +dist_libconfig_DATA = \
11 + ebpf_process.conf \
12 + $(NULL)
\ No newline at end of file
collectors/ebpf_process.plugin/README.md new
+165
@@ -0,0 +1,165 @@
1 +# ebpf_process.plugin
2 +
3 +This plugin uses eBPF to monitor system calls inside your operating system's kernel. For now, the main goal of this
4 +plugin is to monitor IO and process management on the host where it is running.
5 +
6 +This plugin has different configuration modes, all of which can be adjusted with its configuration file at
7 +`ebpf_process.conf`. By default, the plugin uses the less expensive `entry` mode. You can learn more about how the
8 +plugin works using `entry` by reading this configuration file.
9 +
10 +You can always edit this file with `edit-config`:
11 +
12 +```bash
13 +cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/
14 +./edit-config ebpf_process.conf
15 +```
16 +
17 +## Enable the plugin on Linux
18 +
19 +Currently, `ebpf_process` only works on Linux systems.
20 +
21 +To enable this plugin and its collector, your operating system's kernel must be more recent than `4.11.0`, and it must
22 +be compiled with the option `CONFIG_KPROBES=y`. You can verify whether your kernel has this option enabled by running
23 +the following commands:
24 +
25 +```bash
26 +# grep CONFIG_KPROBES=y /boot/config-$(uname -r)
27 +# zgrep CONFIG_KPROBES=y /proc/config.gz
28 +```
29 +
30 +If `Kprobes` is enabled, you will see `CONFIG_KPROBES=y` as the command's output. If you don't see `CONFIG_KPROBES=y`
31 +for any of the commands above, you will have to recompile your kernel to enable it. See the next step, [Recompiling your
32 +kernel](#recompile-your-kernel), for details.
33 +
34 +You also need to have both the `tracefs` and `debugfs` filesystems mounted on your system.
35 +
36 +### Recompile your kernel
37 +
38 +The process of recompiling Linux kernels varies based on your distribution and version. Read the documentation for your
39 +system's distribution to learn more about the specific workflow for
40 +
41 +- [Ubuntu](https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel)
42 +- [Debian](https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-common-official)
43 +- [Fedora](https://fedoraproject.org/wiki/Building_a_custom_kernel)
44 +- [CentOS](https://wiki.centos.org/HowTos/Custom_Kernel)
45 +- [Arch Linux](https://wiki.archlinux.org/index.php/Kernel/Traditional_compilation)
46 +- [Slackware](https://docs.slackware.com/howtos:slackware_admin:kernelbuilding)
47 +
48 +### Mount `debugfs` and `tracefs`
49 +
50 +Try mounting the `tracefs` and `debugfs` filesystems using the commands below:
51 +
52 +```bash
53 +# mount -t debugfs nodev /sys/kernel/debug
54 +# mount -t tracefs nodev /sys/kernel/tracing
55 +```
56 +​
57 +If they are already mounted, you will see an error. You can also configure your system's `/etc/fstab` configuration to
58 +mount these filesystems.
59 +
60 +## Enable the eBPF plugin
61 +The plugin is disabled by default because it adds overhead to the system running the Netdata agent.
62 +
63 +To enable it, use `edit-config` to open `netdata.conf` and set `ebpf_process = yes` in the `[plugins]` section.
64 +
65 +```conf
66 +[plugins]
67 + ebpf_process = yes
68 +```
69 +
70 +## Charts
71 +
72 +The first version of `ebpf_process.plugin` gives a general vision about process running on computer. The charts related
73 +to this plugin are inside the **eBPF** option on dashboard menu and divided in three groups `file`, `vfs`, and
74 +`process`.
75 +
76 +All the collector charts show values per second. The collector retains the total value, but charts only show the
77 +difference between the previous and current metrics collections.
78 +
79 +### File
80 +
81 +This group has two charts to demonstrate how software interacts with the Linux kernel to open and close file
82 +descriptors.
83 +
84 +#### File descriptor
85 +
86 +This chart contain two dimensions that show the number of calls to the functions `do_sys_open` and `__close_fd`. These
87 +functions are not commonly called from software, but they are behind the system cals `open(2)`, `openat(2)`, and
88 +`close(2)`. ​
89 +
90 +#### File error
91 +
92 +This charts demonstrate the number of times some software tried and failed to open or close a file descriptor.
93 +
94 +### VFS
95 +
96 +A [virtual file system](https://en.wikipedia.org/wiki/Virtual_file_system) (VFS) is an layer on top of regular
97 +filesystems. The functions present inside this API are used for all filesystems, so it's possible the charts in this
98 +group won't show _all_ the actions that occured on your system.
99 +
100 +#### Deleted objects
101 +
102 +This chart monitors calls for `vfs_unlink`. This function is responsible for removing object from the file system.
103 +
104 +#### IO
105 +
106 +This chart shows the number of calls to the functions `vfs_read` and `vfs_write`.
107 +
108 +#### IO bytes
109 +
110 +This chart also monitors `vfs_read` and `vfs_write`, but instead shows the total of bytes read and written with these
111 +functions.
112 +
113 +Netdata displays the number of bytes written as negative, because they are moving down to disk.
114 +
115 +#### IO errors
116 +
117 +Netdata counts and shows the number of instances where a running program experiences a read or write error.
118 +
119 +### Process
120 +
121 +For this group, the eBPF collector monitors process/thread creation and process end, and then displays any errors in the
122 +following charts.
123 +
124 +#### Process thread
125 +
126 +Internally, the Linux kernel treats both process and threads as `tasks`. To create a thread, the kernel offers a few
127 +system calls: `fork(2)`, `vfork(2)` and `clone(2)`. Each of these system calls in turn use the function `_do_fork`. To
128 +generate this chart, Netdata monitors `_do_fork` to populate the `process` dimension, and monitors `sys_clone` to
129 +identify threads
130 +
131 +#### Exit
132 +
133 +Ending a task is actually two steps. The first is a call to the internal function `do_exit`, which notifies the
134 +operating system that the task is finishing its work. The second step is the release of kernel information, which is
135 +done with the internal function `release_task`. The difference between the two dimensions can help you discover [zombie
136 +processes](https://en.wikipedia.org/wiki/Zombie_process).
137 +
138 +#### Task error
139 +
140 +The functions responsible for ending tasks do not return values, so this chart contains information about failures on
141 +process and thread creation.
142 +
143 +## Configuration
144 +
145 +The collector configuration file follows the same structure as `netdata.conf`. It is divided in different sections, with
146 +each one of them having the internal variables.
147 +
148 +### `[global]`
149 +
150 +In this section we define variables applied to the whole collector and the other subsections.
151 +
152 +#### load
153 +
154 +The collector has three different eBPF programs. These programs monitor the same functions inside the kernel, but they
155 +monitor, process, and display different kinds of information.
156 +
157 +By default, this plugin uses the `entry` mode. Changing this mode can create significant overhead on your operating
158 +system, but also offer important information if you are developing or debugging software. The `load` option accepts the
159 +following values: ​
160 +
161 +- `entry`: This is the default mode. In this mode, Netdata monitors only calls for the functions described in the
162 + sections above. When this mode is selected, Netdata does not show charts related to errors.
163 +- `return`: In this mode, Netdata also monitors the calls to function. In the `entry` mode, Netdata only traces kernel
164 + functions, but with `return`, Netdata also monitors the return of each function. This mode creates more charts, but
165 + also creates an overhead of roughly 110 nanosections for each function call.
collectors/ebpf_process.plugin/ebpf_process.c new
+948
@@ -0,0 +1,948 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include <sys/time.h>
4 +#include <sys/resource.h>
5 +
6 +#include "ebpf_process.h"
7 +
8 +// callback required by eval()
9 +int health_variable_lookup(const char *variable, uint32_t hash, struct rrdcalc *rc, calculated_number *result) {
10 + (void)variable;
11 + (void)hash;
12 + (void)rc;
13 + (void)result;
14 + return 0;
15 +};
16 +
17 +void send_statistics( const char *action, const char *action_result, const char *action_data) {
18 + (void) action;
19 + (void) action_result;
20 + (void) action_data;
21 + return;
22 +}
23 +
24 +// callbacks required by popen()
25 +void signals_block(void) {};
26 +void signals_unblock(void) {};
27 +void signals_reset(void) {};
28 +
29 +// required by get_system_cpus()
30 +char *netdata_configured_host_prefix = "";
31 +
32 +// callback required by fatal()
33 +void netdata_cleanup_and_exit(int ret) {
34 + exit(ret);
35 +}
36 +
37 +// ----------------------------------------------------------------------
38 +//Netdata eBPF library
39 +void *libnetdata = NULL;
40 +int (*load_bpf_file)(char *, int) = NULL;
41 +int (*set_bpf_perf_event)(int, int);
42 +int (*perf_event_unmap)(struct perf_event_mmap_page *, size_t);
43 +int (*perf_event_mmap_header)(int, struct perf_event_mmap_page **, int);
44 +void (*netdata_perf_loop_multi)(int *, struct perf_event_mmap_page **, int, int *, int (*nsb)(void *, int), int);
45 +int *map_fd = NULL;
46 +
47 +//Perf event variables
48 +static int pmu_fd[NETDATA_MAX_PROCESSOR];
49 +static struct perf_event_mmap_page *headers[NETDATA_MAX_PROCESSOR];
50 +int page_cnt = 8;
51 +
52 +//Libbpf (It is necessary to have at least kernel 4.10)
53 +int (*bpf_map_lookup_elem)(int, const void *, void *);
54 +
55 +static char *plugin_dir = PLUGINS_DIR;
56 +static char *user_config_dir = CONFIG_DIR;
57 +static char *stock_config_dir = LIBCONFIG_DIR;
58 +static char *netdata_configured_log_dir = LOG_DIR;
59 +
60 +FILE *developer_log = NULL;
61 +
62 +//Global vectors
63 +netdata_syscall_stat_t *aggregated_data = NULL;
64 +netdata_publish_syscall_t *publish_aggregated = NULL;
65 +
66 +static int update_every = 1;
67 +static int thread_finished = 0;
68 +static int close_plugin = 0;
69 +static int mode = 2;
70 +static int debug_log = 0;
71 +static int use_stdout = 0;
72 +struct config collector_config;
73 +static int mykernel = 0;
74 +static int nprocs;
75 +uint32_t *hash_values;
76 +
77 +pthread_mutex_t lock;
78 +
79 +static char *dimension_names[NETDATA_MAX_MONITOR_VECTOR] = { "open", "close", "delete", "read", "write", "process", "task", "process", "thread" };
80 +static char *id_names[NETDATA_MAX_MONITOR_VECTOR] = { "do_sys_open", "__close_fd", "vfs_unlink", "vfs_read", "vfs_write", "do_exit", "release_task", "_do_fork", "sys_clone" };
81 +static char *status[] = { "process", "zombie" };
82 +
83 +int event_pid = 0;
84 +netdata_ebpf_events_t collector_events[] = {
85 + { .type = 'r', .name = "vfs_write" },
86 + { .type = 'r', .name = "vfs_writev" },
87 + { .type = 'r', .name = "vfs_read" },
88 + { .type = 'r', .name = "vfs_readv" },
89 + { .type = 'r', .name = "do_sys_open" },
90 + { .type = 'r', .name = "vfs_unlink" },
91 + { .type = 'p', .name = "do_exit" },
92 + { .type = 'p', .name = "release_task" },
93 + { .type = 'r', .name = "_do_fork" },
94 + { .type = 'r', .name = "__close_fd" },
95 + { .type = 'r', .name = "__x64_sys_clone" },
96 + { .type = 0, .name = NULL }
97 +};
98 +
99 +void open_developer_log() {
100 + char filename[FILENAME_MAX+1];
101 + int tot = sprintf(filename, "%s/%s", netdata_configured_log_dir, NETDATA_DEVELOPER_LOG_FILE);
102 +
103 + if(tot > 0)
104 + developer_log = fopen(filename, "a");
105 +}
106 +
107 +static int unmap_memory() {
108 + int i;
109 + int size = (int)sysconf(_SC_PAGESIZE)*(page_cnt + 1);
110 + for ( i = 0 ; i < nprocs ; i++ ) {
111 + if (perf_event_unmap(headers[i], size) < 0) {
112 + fprintf(stderr,"[EBPF PROCESS] CANNOT unmap headers.\n");
113 + return -1;
114 + }
115 +
116 + close(pmu_fd[i]);
117 + }
118 +
119 + return 0;
120 +}
121 +
122 +static void int_exit(int sig)
123 +{
124 + close_plugin = 1;
125 +
126 + //When both threads were not finished case I try to go in front this address, the collector will crash
127 + if (!thread_finished) {
128 + return;
129 + }
130 +
131 + if (aggregated_data) {
132 + free(aggregated_data);
133 + aggregated_data = NULL;
134 + }
135 +
136 + if (publish_aggregated) {
137 + free(publish_aggregated);
138 + publish_aggregated = NULL;
139 + }
140 +
141 + if(mode == 1 && debug_log) {
142 + unmap_memory();
143 + }
144 +
145 + if (libnetdata) {
146 + dlclose(libnetdata);
147 + libnetdata = NULL;
148 + }
149 +
150 + if (developer_log) {
151 + fclose(developer_log);
152 + developer_log = NULL;
153 + }
154 +
155 + if (hash_values) {
156 + freez(hash_values);
157 + }
158 +
159 + if (event_pid) {
160 + int ret = fork();
161 + if (ret < 0) //error
162 + error("[EBPF PROCESS] Cannot fork(), so I won't be able to clean %skprobe_events", NETDATA_DEBUGFS);
163 + else if (!ret) { //child
164 + int i;
165 + for ( i=getdtablesize(); i>=0; --i)
166 + close(i);
167 +
168 + int fd = open("/dev/null",O_RDWR, 0);
169 + if (fd != -1) {
170 + dup2 (fd, STDIN_FILENO);
171 + dup2 (fd, STDOUT_FILENO);
172 + dup2 (fd, STDERR_FILENO);
173 +
174 + if (fd > 2)
175 + close (fd);
176 + }
177 +
178 + int sid = setsid();
179 + if(sid >= 0) {
180 + sleep(1);
181 + if(debug_log) {
182 + open_developer_log();
183 + }
184 + debug(D_EXIT, "Wait for father %d die", event_pid);
185 + clean_kprobe_events(developer_log, event_pid, collector_events);
186 + } else {
187 + error("Cannot become session id leader, so I won't try to clean kprobe_events.\n");
188 + }
189 + } else { //parent
190 + exit(0);
191 + }
192 +
193 + if (developer_log) {
194 + fclose(developer_log);
195 + developer_log = NULL;
196 + }
197 + }
198 +
199 + exit(sig);
200 +}
201 +
202 +static inline void netdata_write_chart_cmd(char *type
203 + , char *id
204 + , char *axis
205 + , char *web
206 + , int order)
207 +{
208 + printf("CHART %s.%s '' '' '%s' '%s' '' line %d 1 ''\n"
209 + , type
210 + , id
211 + , axis
212 + , web
213 + , order);
214 +}
215 +
216 +static void netdata_write_global_dimension(char *dimension, char *name)
217 +{
218 + printf("DIMENSION %s %s absolute 1 1\n", dimension, name);
219 +}
220 +
221 +static void netdata_create_global_dimension(void *ptr, int end)
222 +{
223 + netdata_publish_syscall_t *move = ptr;
224 +
225 + int i = 0;
226 + while (move && i < end) {
227 + netdata_write_global_dimension(move->name, move->dimension);
228 +
229 + move = move->next;
230 + i++;
231 + }
232 +}
233 +static inline void netdata_create_chart(char *family
234 + , char *name
235 + , char *axis
236 + , char *web
237 + , int order
238 + , void (*ncd)(void *, int)
239 + , void *move
240 + , int end)
241 +{
242 + netdata_write_chart_cmd(family, name, axis, web, order);
243 +
244 + ncd(move, end);
245 +}
246 +
247 +static void netdata_create_io_chart(char *family, char *name, char *axis, char *web, int order) {
248 + printf("CHART %s.%s '' '' '%s' '%s' '' line %d 1 ''\n"
249 + , family
250 + , name
251 + , axis
252 + , web
253 + , order);
254 +
255 + printf("DIMENSION %s %s absolute 1 1\n", id_names[3], NETDATA_VFS_DIM_OUT_FILE_BYTES);
256 + printf("DIMENSION %s %s absolute 1 1\n", id_names[4], NETDATA_VFS_DIM_IN_FILE_BYTES);
257 +}
258 +
259 +static void netdata_process_status_chart(char *family, char *name, char *axis, char *web, int order) {
260 + printf("CHART %s.%s '' '' '%s' '%s' '' line %d 1 ''\n"
261 + , family
262 + , name
263 + , axis
264 + , web
265 + , order);
266 +
267 + printf("DIMENSION %s '' absolute 1 1\n", status[0]);
268 + printf("DIMENSION %s '' absolute 1 1\n", status[1]);
269 +}
270 +
271 +static void netdata_global_charts_create() {
272 + netdata_create_chart(NETDATA_EBPF_FAMILY
273 + , NETDATA_FILE_OPEN_CLOSE_COUNT
274 + , "Calls"
275 + , NETDATA_FILE_GROUP
276 + , 970
277 + , netdata_create_global_dimension
278 + , publish_aggregated
279 + , 2);
280 +
281 + if(mode < 2) {
282 + netdata_create_chart(NETDATA_EBPF_FAMILY
283 + , NETDATA_FILE_OPEN_ERR_COUNT
284 + , "Calls"
285 + , NETDATA_FILE_GROUP
286 + , 971
287 + , netdata_create_global_dimension
288 + , publish_aggregated
289 + , 2);
290 + }
291 +
292 + netdata_create_chart(NETDATA_EBPF_FAMILY
293 + , NETDATA_VFS_FILE_CLEAN_COUNT
294 + , "Calls"
295 + , NETDATA_VFS_GROUP
296 + , 972
297 + , netdata_create_global_dimension
298 + , &publish_aggregated[NETDATA_DEL_START]
299 + , 1);
300 +
301 + netdata_create_chart(NETDATA_EBPF_FAMILY
302 + , NETDATA_VFS_FILE_IO_COUNT
303 + , "Calls"
304 + , NETDATA_VFS_GROUP
305 + , 973
306 + , netdata_create_global_dimension
307 + , &publish_aggregated[NETDATA_IN_START_BYTE]
308 + , 2);
309 +
310 + if(mode < 2) {
311 + netdata_create_io_chart(NETDATA_EBPF_FAMILY
312 + , NETDATA_VFS_IO_FILE_BYTES
313 + , "bytes/s"
314 + , NETDATA_VFS_GROUP
315 + , 974);
316 +
317 + netdata_create_chart(NETDATA_EBPF_FAMILY
318 + , NETDATA_VFS_FILE_ERR_COUNT
319 + , "Calls"
320 + , NETDATA_VFS_GROUP
321 + , 975
322 + , netdata_create_global_dimension
323 + , &publish_aggregated[2]
324 + , NETDATA_VFS_ERRORS);
325 +
326 + }
327 +
328 + netdata_create_chart(NETDATA_EBPF_FAMILY
329 + , NETDATA_PROCESS_SYSCALL
330 + , "Calls"
331 + , NETDATA_PROCESS_GROUP
332 + , 976
333 + , netdata_create_global_dimension
334 + , &publish_aggregated[NETDATA_PROCESS_START]
335 + , 2);
336 +
337 + netdata_create_chart(NETDATA_EBPF_FAMILY
338 + , NETDATA_EXIT_SYSCALL
339 + , "Calls"
340 + , NETDATA_PROCESS_GROUP
341 + , 977
342 + , netdata_create_global_dimension
343 + , &publish_aggregated[NETDATA_EXIT_START]
344 + , 2);
345 +
346 + netdata_process_status_chart(NETDATA_EBPF_FAMILY
347 + , NETDATA_PROCESS_STATUS_NAME
348 + , "Total"
349 + , NETDATA_PROCESS_GROUP
350 + , 978);
351 +
352 + if(mode < 2) {
353 + netdata_create_chart(NETDATA_EBPF_FAMILY
354 + , NETDATA_PROCESS_ERROR_NAME
355 + , "Calls"
356 + , NETDATA_PROCESS_GROUP
357 + , 979
358 + , netdata_create_global_dimension
359 + , &publish_aggregated[NETDATA_PROCESS_START]
360 + , 2);
361 + }
362 +
363 +}
364 +
365 +
366 +static void netdata_create_charts() {
367 + netdata_global_charts_create();
368 +}
369 +
370 +static void netdata_update_publish(netdata_publish_syscall_t *publish
371 + , netdata_publish_vfs_common_t *pvc
372 + , netdata_syscall_stat_t *input) {
373 +
374 + netdata_publish_syscall_t *move = publish;
375 + while(move) {
376 + if(input->call != move->pcall) {
377 + //This condition happens to avoid initial values with dimensions higher than normal values.
378 + if(move->pcall) {
379 + move->ncall = (input->call > move->pcall)?input->call - move->pcall: move->pcall - input->call;
380 + move->nbyte = (input->bytes > move->pbyte)?input->bytes - move->pbyte: move->pbyte - input->bytes;
381 + move->nerr = (input->ecall > move->nerr)?input->ecall - move->perr: move->perr - input->ecall;
382 + } else {
383 + move->ncall = 0;
384 + move->nbyte = 0;
385 + move->nerr = 0;
386 + }
387 +
388 + move->pcall = input->call;
389 + move->pbyte = input->bytes;
390 + move->perr = input->ecall;
391 + } else {
392 + move->ncall = 0;
393 + move->nbyte = 0;
394 + move->nerr = 0;
395 + }
396 +
397 + input = input->next;
398 + move = move->next;
399 + }
400 +
401 + pvc->write = -((long)publish[2].nbyte);
402 + pvc->read = (long)publish[3].nbyte;
403 +
404 + pvc->running = (long)publish[7].ncall - (long)publish[8].ncall;
405 + publish[6].ncall = -publish[6].ncall; // release
406 + pvc->zombie = (long)publish[5].ncall + (long)publish[6].ncall;
407 +}
408 +
409 +static inline void write_begin_chart(char *family, char *name)
410 +{
411 + int ret = printf( "BEGIN %s.%s\n"
412 + , family
413 + , name);
414 +
415 + (void)ret;
416 +}
417 +
418 +static inline void write_chart_dimension(char *dim, long long value)
419 +{
420 + int ret = printf("SET %s = %lld\n", dim, value);
421 + (void)ret;
422 +}
423 +
424 +static void write_global_count_chart(char *name, char *family, netdata_publish_syscall_t *move, int end) {
425 + write_begin_chart(family, name);
426 +
427 + int i = 0;
428 + while (move && i < end) {
429 + write_chart_dimension(move->name, move->ncall);
430 +
431 + move = move->next;
432 + i++;
433 + }
434 +
435 + printf("END\n");
436 +}
437 +
438 +static void write_global_err_chart(char *name, char *family, netdata_publish_syscall_t *move, int end) {
439 + write_begin_chart(family, name);
440 +
441 + int i = 0;
442 + while (move && i < end) {
443 + write_chart_dimension(move->name, move->nerr);
444 +
445 + move = move->next;
446 + i++;
447 + }
448 +
449 + printf("END\n");
450 +}
451 +
452 +static void write_io_chart(char *family, netdata_publish_vfs_common_t *pvc) {
453 + write_begin_chart(family, NETDATA_VFS_IO_FILE_BYTES);
454 +
455 + write_chart_dimension(id_names[3], (long long) pvc->write);
456 + write_chart_dimension(id_names[4], (long long) pvc->read);
457 +
458 + printf("END\n");
459 +}
460 +
461 +static void write_status_chart(char *family, netdata_publish_vfs_common_t *pvc) {
462 + write_begin_chart(family, NETDATA_PROCESS_STATUS_NAME);
463 +
464 + write_chart_dimension(status[0], (long long) pvc->running);
465 + write_chart_dimension(status[1], (long long) pvc->zombie);
466 +
467 + printf("END\n");
468 +}
469 +
470 +static void netdata_publish_data() {
471 + netdata_publish_vfs_common_t pvc;
472 + netdata_update_publish(publish_aggregated, &pvc, aggregated_data);
473 +
474 + write_global_count_chart(NETDATA_FILE_OPEN_CLOSE_COUNT, NETDATA_EBPF_FAMILY, publish_aggregated, 2);
475 + write_global_count_chart(NETDATA_VFS_FILE_CLEAN_COUNT, NETDATA_EBPF_FAMILY, &publish_aggregated[NETDATA_DEL_START], 1);
476 + write_global_count_chart(NETDATA_VFS_FILE_IO_COUNT, NETDATA_EBPF_FAMILY, &publish_aggregated[NETDATA_IN_START_BYTE], 2);
477 + write_global_count_chart(NETDATA_EXIT_SYSCALL, NETDATA_EBPF_FAMILY, &publish_aggregated[NETDATA_EXIT_START], 2);
478 + write_global_count_chart(NETDATA_PROCESS_SYSCALL, NETDATA_EBPF_FAMILY, &publish_aggregated[NETDATA_PROCESS_START], 2);
479 +
480 + write_status_chart(NETDATA_EBPF_FAMILY, &pvc);
481 + if(mode < 2) {
482 + write_global_err_chart(NETDATA_FILE_OPEN_ERR_COUNT, NETDATA_EBPF_FAMILY, publish_aggregated, 2);
483 + write_global_err_chart(NETDATA_VFS_FILE_ERR_COUNT, NETDATA_EBPF_FAMILY, &publish_aggregated[2], NETDATA_VFS_ERRORS);
484 + write_global_err_chart(NETDATA_PROCESS_ERROR_NAME, NETDATA_EBPF_FAMILY, &publish_aggregated[NETDATA_PROCESS_START], 2);
485 +
486 + write_io_chart(NETDATA_EBPF_FAMILY, &pvc);
487 + }
488 +}
489 +
490 +void *process_publisher(void *ptr)
491 +{
492 + (void)ptr;
493 + netdata_create_charts();
494 +
495 + usec_t step = update_every * USEC_PER_SEC;
496 + heartbeat_t hb;
497 + heartbeat_init(&hb);
498 + while(!close_plugin) {
499 + usec_t dt = heartbeat_next(&hb, step);
500 + (void)dt;
501 +
502 + pthread_mutex_lock(&lock);
503 + netdata_publish_data();
504 + pthread_mutex_unlock(&lock);
505 +
506 + fflush(stdout);
507 + }
508 +
509 + return NULL;
510 +}
511 +
512 +static void move_from_kernel2user_global() {
513 + uint32_t idx;
514 + uint32_t res[NETDATA_GLOBAL_VECTOR];
515 +
516 + uint32_t *val = hash_values;
517 + for (idx = 0; idx < NETDATA_GLOBAL_VECTOR; idx++) {
518 + if(!bpf_map_lookup_elem(map_fd[1], &idx, val)) {
519 + uint32_t total = 0;
520 + int i;
521 + int end = (mykernel < 265984)?1:nprocs;
522 + for (i = 0; i < end; i++)
523 + total += val[i];
524 +
525 + res[idx] = total;
526 + } else {
527 + res[idx] = 0;
528 + }
529 + }
530 +
531 + aggregated_data[0].call = res[0]; //open
532 + aggregated_data[1].call = res[14]; //close
533 + aggregated_data[2].call = res[8]; //unlink
534 + aggregated_data[3].call = res[5] + res[21]; //read + readv
535 + aggregated_data[4].call = res[2] + res[18]; //write + writev
536 + aggregated_data[5].call = res[10]; //exit
537 + aggregated_data[6].call = res[11]; //release
538 + aggregated_data[7].call = res[12]; //fork
539 + aggregated_data[8].call = res[16]; //thread
540 +
541 + aggregated_data[0].ecall = res[1]; //open
542 + aggregated_data[1].ecall = res[15]; //close
543 + aggregated_data[2].ecall = res[9]; //unlink
544 + aggregated_data[3].ecall = res[6] + res[22]; //read + readv
545 + aggregated_data[4].ecall = res[3] + res[19]; //write + writev
546 + aggregated_data[7].ecall = res[13]; //fork
547 + aggregated_data[8].ecall = res[17]; //thread
548 +
549 + aggregated_data[2].bytes = (uint64_t)res[4] + (uint64_t)res[20]; //write + writev
550 + aggregated_data[3].bytes = (uint64_t)res[7] + (uint64_t)res[23];//read + readv
551 +}
552 +
553 +static void move_from_kernel2user()
554 +{
555 + move_from_kernel2user_global();
556 +}
557 +
558 +void *process_collector(void *ptr)
559 +{
560 + (void)ptr;
561 +
562 + usec_t step = 778879ULL;
563 + heartbeat_t hb;
564 + heartbeat_init(&hb);
565 + while(!close_plugin) {
566 + usec_t dt = heartbeat_next(&hb, step);
567 + (void)dt;
568 +
569 + pthread_mutex_lock(&lock);
570 + move_from_kernel2user();
571 + pthread_mutex_unlock(&lock);
572 + }
573 +
574 + return NULL;
575 +}
576 +
577 +static int netdata_store_bpf(void *data, int size) {
578 + (void)size;
579 +
580 + if (close_plugin)
581 + return 0;
582 +
583 + if(!debug_log)
584 + return -2; //LIBBPF_PERF_EVENT_CONT;
585 +
586 + netdata_error_report_t *e = data;
587 + fprintf(developer_log
588 + ,"%llu %s %u: %s, %d\n"
589 + , now_realtime_usec() ,e->comm, e->pid, dimension_names[e->type], e->err);
590 + fflush(developer_log);
591 +
592 + return -2; //LIBBPF_PERF_EVENT_CONT;
593 +}
594 +
595 +void *process_log(void *ptr)
596 +{
597 + (void) ptr;
598 +
599 + if (mode == 1 && debug_log) {
600 + netdata_perf_loop_multi(pmu_fd, headers, nprocs, &close_plugin, netdata_store_bpf, page_cnt);
601 + }
602 +
603 + return NULL;
604 +}
605 +
606 +void set_global_labels() {
607 + int i;
608 +
609 + netdata_syscall_stat_t *is = aggregated_data;
610 + netdata_syscall_stat_t *prev = NULL;
611 +
612 + netdata_publish_syscall_t *pio = publish_aggregated;
613 + netdata_publish_syscall_t *publish_prev = NULL;
614 + for (i = 0; i < NETDATA_MAX_MONITOR_VECTOR; i++) {
615 + if(prev) {
616 + prev->next = &is[i];
617 + }
618 + prev = &is[i];
619 +
620 + pio[i].dimension = dimension_names[i];
621 + pio[i].name = id_names[i];
622 + if(publish_prev) {
623 + publish_prev->next = &pio[i];
624 + }
625 + publish_prev = &pio[i];
626 + }
627 +}
628 +
629 +int allocate_global_vectors() {
630 + aggregated_data = callocz(NETDATA_MAX_MONITOR_VECTOR, sizeof(netdata_syscall_stat_t));
631 + if(!aggregated_data) {
632 + return -1;
633 + }
634 +
635 + publish_aggregated = callocz(NETDATA_MAX_MONITOR_VECTOR, sizeof(netdata_publish_syscall_t));
636 + if(!publish_aggregated) {
637 + return -1;
638 + }
639 +
640 + hash_values = callocz(nprocs, sizeof(uint32_t));
641 + if(!hash_values) {
642 + return -1;
643 + }
644 +
645 + return 0;
646 +}
647 +
648 +static void build_complete_path(char *out, size_t length,char *path, char *filename) {
649 + if(path){
650 + snprintf(out, length, "%s/%s", path, filename);
651 + } else {
652 + snprintf(out, length, "%s", filename);
653 + }
654 +}
655 +
656 +static int map_memory() {
657 + int i;
658 + for (i = 0; i < nprocs; i++) {
659 + pmu_fd[i] = set_bpf_perf_event(i, 2);
660 +
661 + if (perf_event_mmap_header(pmu_fd[i], &headers[i], page_cnt) < 0) {
662 + return -1;
663 + }
664 + }
665 + return 0;
666 +}
667 +
668 +static int ebpf_load_libraries()
669 +{
670 + char *err = NULL;
671 + char lpath[4096];
672 +
673 + build_complete_path(lpath, 4096, plugin_dir, "libnetdata_ebpf.so");
674 + libnetdata = dlopen(lpath, RTLD_LAZY);
675 + if (!libnetdata) {
676 + error("[EBPF_PROCESS] Cannot load %s.", lpath);
677 + return -1;
678 + } else {
679 + load_bpf_file = dlsym(libnetdata, "load_bpf_file");
680 + if ((err = dlerror()) != NULL) {
681 + error("[EBPF_PROCESS] Cannot find load_bpf_file: %s", err);
682 + return -1;
683 + }
684 +
685 + map_fd = dlsym(libnetdata, "map_fd");
686 + if ((err = dlerror()) != NULL) {
687 + error("[EBPF_PROCESS] Cannot find map_fd: %s", err);
688 + return -1;
689 + }
690 +
691 + bpf_map_lookup_elem = dlsym(libnetdata, "bpf_map_lookup_elem");
692 + if ((err = dlerror()) != NULL) {
693 + error("[EBPF_PROCESS] Cannot find bpf_map_lookup_elem: %s", err);
694 + return -1;
695 + }
696 +
697 + if(mode == 1) {
698 + set_bpf_perf_event = dlsym(libnetdata, "set_bpf_perf_event");
699 + if ((err = dlerror()) != NULL) {
700 + error("[EBPF_PROCESS] Cannot find set_bpf_perf_event: %s", err);
701 + return -1;
702 + }
703 +
704 + perf_event_unmap = dlsym(libnetdata, "perf_event_unmap");
705 + if ((err = dlerror()) != NULL) {
706 + error("[EBPF_PROCESS] Cannot find perf_event_unmap: %s", err);
707 + return -1;
708 + }
709 +
710 + perf_event_mmap_header = dlsym(libnetdata, "perf_event_mmap_header");
711 + if ((err = dlerror()) != NULL) {
712 + error("[EBPF_PROCESS] Cannot find perf_event_mmap_header: %s", err);
713 + return -1;
714 + }
715 +
716 + netdata_perf_loop_multi = dlsym(libnetdata, "netdata_perf_loop_multi");
717 + if ((err = dlerror()) != NULL) {
718 + error("[EBPF_PROCESS] Cannot find netdata_perf_loop_multi: %s", err);
719 + return -1;
720 + }
721 + }
722 + }
723 +
724 + return 0;
725 +}
726 +
727 +char *select_file() {
728 + if(!mode)
729 + return "rnetdata_ebpf_process.o";
730 + if(mode == 1)
731 + return "dnetdata_ebpf_process.o";
732 +
733 + return "pnetdata_ebpf_process.o";
734 +}
735 +
736 +int process_load_ebpf()
737 +{
738 + char lpath[4096];
739 +
740 + char *name = select_file();
741 +
742 + build_complete_path(lpath, 4096, plugin_dir, name);
743 + event_pid = getpid();
744 + if (load_bpf_file(lpath, event_pid) ) {
745 + error("[EBPF_PROCESS] Cannot load program: %s", lpath);
746 + return -1;
747 + } else {
748 + info("[EBPF PROCESS]: The eBPF program %s was loaded with success.", name);
749 + }
750 +
751 + return 0;
752 +}
753 +
754 +void set_global_variables() {
755 + //Get environment variables
756 + plugin_dir = getenv("NETDATA_PLUGINS_DIR");
757 + if(!plugin_dir)
758 + plugin_dir = PLUGINS_DIR;
759 +
760 + user_config_dir = getenv("NETDATA_USER_CONFIG_DIR");
761 + if(!user_config_dir)
762 + user_config_dir = CONFIG_DIR;
763 +
764 + stock_config_dir = getenv("NETDATA_STOCK_CONFIG_DIR");
765 + if(!stock_config_dir)
766 + stock_config_dir = LIBCONFIG_DIR;
767 +
768 + netdata_configured_log_dir = getenv("NETDATA_LOG_DIR");
769 + if(!netdata_configured_log_dir)
770 + netdata_configured_log_dir = LOG_DIR;
771 +
772 + page_cnt *= (int)sysconf(_SC_NPROCESSORS_ONLN);
773 +
774 + nprocs = (int)sysconf(_SC_NPROCESSORS_ONLN);
775 + if (nprocs > NETDATA_MAX_PROCESSOR) {
776 + nprocs = NETDATA_MAX_PROCESSOR;
777 + }
778 +}
779 +
780 +static void change_collector_event() {
781 + int i;
782 + for (i = 0; collector_events[i].name ; i++ ) {
783 + collector_events[i].type = 'p';
784 + }
785 +
786 + if (mykernel < 328448)
787 + collector_events[i].name = NULL;
788 +}
789 +
790 +static inline void what_to_load(char *ptr) {
791 + if (!strcasecmp(ptr, "return"))
792 + mode = 0;
793 + /*
794 + else if (!strcasecmp(ptr, "dev"))
795 + mode = 1;
796 + */
797 + else
798 + change_collector_event();
799 +}
800 +
801 +static inline void enable_debug(char *ptr) {
802 + if (!strcasecmp(ptr, "yes"))
803 + debug_log = 1;
804 +}
805 +
806 +static inline void set_log_file(char *ptr) {
807 + if (!strcasecmp(ptr, "yes"))
808 + use_stdout = 1;
809 +}
810 +
811 +static void set_global_values() {
812 + struct section *sec = collector_config.sections;
813 + while(sec) {
814 + if(!strcasecmp(sec->name, "global")) {
815 + struct config_option *values = sec->values;
816 + while(values) {
817 + if(!strcasecmp(values->name, "load"))
818 + what_to_load(values->value);
819 + else if(!strcasecmp(values->name, "debug log"))
820 + enable_debug(values->value);
821 + else if(!strcasecmp(values->name, "use stdout"))
822 + set_log_file(values->value);
823 +
824 + values = values->next;
825 + }
826 + }
827 + sec = sec->next;
828 + }
829 +}
830 +
831 +static int load_collector_file(char *path) {
832 + char lpath[4096];
833 +
834 + build_complete_path(lpath, 4096, path, "ebpf_process.conf" );
835 +
836 + if (!appconfig_load(&collector_config, lpath, 0, NULL))
837 + return 1;
838 +
839 + set_global_values();
840 +
841 + return 0;
842 +}
843 +
844 +int main(int argc, char **argv)
845 +{
846 + (void)argc;
847 + (void)argv;
848 +
849 + mykernel = get_kernel_version();
850 + if(!has_condition_to_run(mykernel))
851 + return 1;
852 +
853 + //set name
854 + program_name = "ebpf_process.plugin";
855 +
856 + //disable syslog
857 + error_log_syslog = 0;
858 +
859 + // set errors flood protection to 100 logs per hour
860 + error_log_errors_per_period = 100;
861 + error_log_throttle_period = 3600;
862 +
863 + if (argc > 1) {
864 + update_every = (int)strtol(argv[1], NULL, 10);
865 + }
866 +
867 + struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
868 + if (setrlimit(RLIMIT_MEMLOCK, &r)) {
869 + error("[EBPF PROCESS] setrlimit(RLIMIT_MEMLOCK)");
870 + return 2;
871 + }
872 +
873 + set_global_variables();
874 +
875 + if (load_collector_file(user_config_dir)) {
876 + info("[EBPF PROCESS] does not have a configuration file. It is starting with default options.");
877 + }
878 +
879 + if(ebpf_load_libraries()) {
880 + error("[EBPF_PROCESS] Cannot load library.");
881 + thread_finished++;
882 + int_exit(3);
883 + }
884 +
885 + signal(SIGINT, int_exit);
886 + signal(SIGTERM, int_exit);
887 +
888 + if (process_load_ebpf()) {
889 + thread_finished++;
890 + int_exit(4);
891 + }
892 +
893 + if(allocate_global_vectors()) {
894 + thread_finished++;
895 + error("[EBPF_PROCESS] Cannot allocate necessary vectors.");
896 + int_exit(5);
897 + }
898 +
899 + if(mode == 1 && debug_log) {
900 + if(map_memory()) {
901 + thread_finished++;
902 + error("[EBPF_PROCESS] Cannot map memory used with perf events.");
903 + int_exit(6);
904 + }
905 + }
906 +
907 + set_global_labels();
908 +
909 + if(debug_log) {
910 + open_developer_log();
911 + }
912 +
913 + if (pthread_mutex_init(&lock, NULL)) {
914 + thread_finished++;
915 + int_exit(7);
916 + }
917 +
918 + pthread_attr_t attr;
919 + pthread_attr_init(&attr);
920 + pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);
921 + pthread_t thread[NETDATA_EBPF_PROCESS_THREADS];
922 +
923 + int i;
924 + int end = NETDATA_EBPF_PROCESS_THREADS;
925 +
926 + void * (*function_pointer[])(void *) = {process_publisher, process_collector, process_log };
927 +
928 + for ( i = 0; i < end ; i++ ) {
929 + if ( ( pthread_create(&thread[i], &attr, function_pointer[i], NULL) ) ) {
930 + error("[EBPF_PROCESS] Cannot create threads.");
931 + thread_finished++;
932 + int_exit(8);
933 + }
934 + }
935 +
936 + for ( i = 0; i < end ; i++ ) {
937 + if ( (pthread_join(thread[i], NULL) ) ) {
938 + error("[EBPF_PROCESS] Cannot join threads.");
939 + thread_finished++;
940 + int_exit(9);
941 + }
942 + }
943 +
944 + thread_finished++;
945 + int_exit(0);
946 +
947 + return 0;
948 +}
collectors/ebpf_process.plugin/ebpf_process.conf new
+4
@@ -0,0 +1,4 @@
1 +[global]
2 + load = entry
3 + debug log = no
4 + use stdout = no
collectors/ebpf_process.plugin/ebpf_process.h new
+103
@@ -0,0 +1,103 @@
1 +#ifndef _NETDATA_VFS_EBPF_H_
2 +# define _NETDATA_VFS_EBPF_H_ 1
3 +
4 +# include <stdint.h>
5 +
6 +#ifndef __FreeBSD__
7 +# include <linux/perf_event.h>
8 +# endif
9 +# include <stdint.h>
10 +# include <errno.h>
11 +# include <signal.h>
12 +# include <stdio.h>
13 +# include <stdint.h>
14 +# include <stdlib.h>
15 +# include <string.h>
16 +# include <unistd.h>
17 +# include <dlfcn.h>
18 +
19 +# define NETDATA_GLOBAL_VECTOR 24
20 +# define NETDATA_MAX_MONITOR_VECTOR 9
21 +# define NETDATA_VFS_ERRORS 3
22 +# define NETDATA_PROCESS_ERRORS 4
23 +
24 +# define NETDATA_DEL_START 2
25 +# define NETDATA_IN_START_BYTE 3
26 +# define NETDATA_EXIT_START 5
27 +# define NETDATA_PROCESS_START 7
28 +# define NETDATA_PROCESS_RUNNING_COUNT 9
29 +
30 +# define NETDATA_EBPF_PROCESS_THREADS (uint32_t)3
31 +
32 +# include <fcntl.h>
33 +# include <ctype.h>
34 +# include <dirent.h>
35 +
36 +//From libnetdata.h
37 +# include "../../libnetdata/threads/threads.h"
38 +# include "../../libnetdata/locks/locks.h"
39 +# include "../../libnetdata/avl/avl.h"
40 +# include "../../libnetdata/clocks/clocks.h"
41 +# include "../../libnetdata/config/appconfig.h"
42 +# include "../../libnetdata/ebpf/ebpf.h"
43 +
44 +typedef struct netdata_syscall_stat {
45 + unsigned long bytes; //total number of bytes
46 + uint64_t call; //total number of calls
47 + uint64_t ecall; //number of calls that returned error
48 + struct netdata_syscall_stat *next; //Link list
49 +}netdata_syscall_stat_t;
50 +
51 +typedef struct netdata_publish_syscall {
52 + char *dimension;
53 + char *name;
54 + unsigned long nbyte;
55 + unsigned long pbyte;
56 + uint64_t ncall;
57 + uint64_t pcall;
58 + uint64_t nerr;
59 + uint64_t perr;
60 + struct netdata_publish_syscall *next;
61 +}netdata_publish_syscall_t;
62 +
63 +typedef struct netdata_publish_vfs_common {
64 + long write;
65 + long read;
66 +
67 + long running;
68 + long zombie;
69 +}netdata_publish_vfs_common_t;
70 +
71 +typedef struct netdata_error_report {
72 + char comm[16];
73 + __u32 pid;
74 +
75 + int type;
76 + int err;
77 +}netdata_error_report_t;
78 +
79 +# define NETDATA_EBPF_FAMILY "ebpf"
80 +# define NETDATA_FILE_GROUP "File"
81 +# define NETDATA_VFS_GROUP "VFS"
82 +# define NETDATA_PROCESS_GROUP "Process"
83 +
84 +# define NETDATA_FILE_OPEN_CLOSE_COUNT "file_descriptor"
85 +# define NETDATA_FILE_OPEN_ERR_COUNT "file_error"
86 +# define NETDATA_VFS_FILE_CLEAN_COUNT "deleted_objects"
87 +# define NETDATA_VFS_FILE_IO_COUNT "io"
88 +# define NETDATA_VFS_FILE_ERR_COUNT "io_error"
89 +
90 +# define NETDATA_EXIT_SYSCALL "exit"
91 +# define NETDATA_PROCESS_SYSCALL "process_thread"
92 +# define NETDATA_PROCESS_ERROR_NAME "task_error"
93 +# define NETDATA_PROCESS_STATUS_NAME "process_status"
94 +
95 +# define NETDATA_VFS_IO_FILE_BYTES "io_bytes"
96 +# define NETDATA_VFS_DIM_IN_FILE_BYTES "write"
97 +# define NETDATA_VFS_DIM_OUT_FILE_BYTES "read"
98 +
99 +# define NETDATA_DEVELOPER_LOG_FILE "developer.log"
100 +
101 +# define NETDATA_MAX_PROCESSOR 512
102 +
103 +#endif
collectors/plugins.d/plugins_d.c
+1
@@ -821,6 +821,7 @@ void *pluginsd_main(void *ptr) {
821
822 // disable some plugins by default
823 config_get_boolean(CONFIG_SECTION_PLUGINS, "slabinfo", CONFIG_BOOLEAN_NO);
824 + config_get_boolean(CONFIG_SECTION_PLUGINS, "ebpf_process", CONFIG_BOOLEAN_NO);
825
826 // store the errno for each plugins directory
827 // so that we don't log broken directories on each loop
configure.ac
+25
@@ -384,6 +384,17 @@ PKG_CHECK_MODULES([JSON],[json-c],AC_CHECK_LIB(
384
385 OPTIONAL_JSONC_LIBS="${JSONC_LIBS}"
386
387 +# -----------------------------------------------------------------------------
388 +# VFS plugin libs
389 +
390 +AC_CHECK_LIB(
391 + [dl],
392 + [dlopen],
393 + [EBPF_LIBS="-ldl"]
394 +)
395 +
396 +OPTIONAL_EBPF_LIBS="${EBPF_LIBS}"
397 +
398 # -----------------------------------------------------------------------------
399 # DB engine and HTTPS
400 test "${enable_dbengine}" = "yes" -a -z "${LZ4_LIBS}" && \
@@ -860,6 +871,17 @@ fi
871 AC_MSG_RESULT([${enable_plugin_perf}])
872 AM_CONDITIONAL([ENABLE_PLUGIN_PERF], [test "${enable_plugin_perf}" = "yes"])
873
874 +# -----------------------------------------------------------------------------
875 +# ebpf_process.plugin
876 +
877 +AC_MSG_CHECKING([if ebpf_process.plugin should be enabled])
878 +if test "${build_target}" == "linux" -a "${EBPF_LIBS}"; then
879 + enable_vfs="yes"
880 +else
881 + enable_vfs="no"
882 +fi
883 +AC_MSG_RESULT([${enable_vfs}])
884 +AM_CONDITIONAL([ENABLE_PLUGIN_EBPF], [test "${enable_vfs}" = "yes"])
885
886 # -----------------------------------------------------------------------------
887 # slabinfo.plugin
@@ -1196,6 +1218,7 @@ AC_SUBST([OPTIONAL_UV_LIBS])
1218 AC_SUBST([OPTIONAL_LZ4_LIBS])
1219 AC_SUBST([OPTIONAL_JUDY_LIBS])
1220 AC_SUBST([OPTIONAL_SSL_LIBS])
1221 +AC_SUBST([OPTIONAL_EBPF_LIBS])
1222 AC_SUBST([OPTIONAL_JSONC_LIBS])
1223 AC_SUBST([OPTIONAL_NFACCT_CFLAGS])
1224 AC_SUBST([OPTIONAL_NFACCT_LIBS])
@@ -1276,6 +1299,7 @@ AC_CONFIG_FILES([
1299 collectors/python.d.plugin/Makefile
1300 collectors/slabinfo.plugin/Makefile
1301 collectors/statsd.plugin/Makefile
1302 + collectors/ebpf_process.plugin/Makefile
1303 collectors/tc.plugin/Makefile
1304 collectors/xenstat.plugin/Makefile
1305 collectors/perf.plugin/Makefile
@@ -1298,6 +1322,7 @@ AC_CONFIG_FILES([
1322 libnetdata/clocks/Makefile
1323 libnetdata/config/Makefile
1324 libnetdata/dictionary/Makefile
1325 + libnetdata/ebpf/Makefile
1326 libnetdata/eval/Makefile
1327 libnetdata/locks/Makefile
1328 libnetdata/log/Makefile
libnetdata/Makefile.am
+1
@@ -10,6 +10,7 @@ SUBDIRS = \
10 clocks \
11 config \
12 dictionary \
13 + ebpf \
14 eval \
15 json \
16 health \
libnetdata/ebpf/Makefile.am new
+8
@@ -0,0 +1,8 @@
1 +# SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +AUTOMAKE_OPTIONS = subdir-objects
4 +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in
5 +
6 +dist_noinst_DATA = \
7 + README.md \
8 + $(NULL)
libnetdata/ebpf/README.md
libnetdata/ebpf/ebpf.c new
+98
@@ -0,0 +1,98 @@
1 +#include <sys/types.h>
2 +#include <sys/stat.h>
3 +#include <fcntl.h>
4 +
5 +#include "../libnetdata.h"
6 +
7 +static int clean_kprobe_event(FILE *out, char *filename, char *father_pid, netdata_ebpf_events_t *ptr) {
8 + int fd = open(filename, O_WRONLY | O_APPEND, 0);
9 + if (fd < 0) {
10 + if(out) {
11 + fprintf(out, "Cannot open %s : %s\n", filename, strerror(errno));
12 + }
13 + return 1;
14 + }
15 +
16 + char cmd[1024];
17 + int length = sprintf(cmd, "-:kprobes/%c_netdata_%s_%s", ptr->type, ptr->name, father_pid);
18 + int ret = 0;
19 + if (length > 0) {
20 + ssize_t written = write(fd, cmd, strlen(cmd));
21 + if (written < 0) {
22 + if(out) {
23 + fprintf(out
24 + , "Cannot remove the event (%d, %d) '%s' from %s : %s\n"
25 + , getppid(), getpid(), cmd, filename, strerror((int)errno));
26 + }
27 + ret = 1;
28 + }
29 + }
30 +
31 + close(fd);
32 +
33 + return ret;
34 +}
35 +
36 +int clean_kprobe_events(FILE *out, int pid, netdata_ebpf_events_t *ptr) {
37 + debug(D_EXIT, "Cleaning parent process events.");
38 + char filename[FILENAME_MAX +1];
39 + snprintf(filename, FILENAME_MAX, "%s%s", NETDATA_DEBUGFS, "kprobe_events");
40 +
41 + char removeme[16];
42 + snprintf(removeme, 15,"%d", pid);
43 +
44 + int i;
45 + for (i = 0 ; ptr[i].name ; i++) {
46 + if (clean_kprobe_event(out, filename, removeme, &ptr[i])) {
47 + break;
48 + }
49 + }
50 +
51 + return 0;
52 +}
53 +
54 +//----------------------------------------------------------------------------------------------------------------------
55 +
56 +int get_kernel_version() {
57 + char major[16], minor[16], patch[16];
58 + char ver[256];
59 + char *version = ver;
60 +
61 + int fd = open("/proc/sys/kernel/osrelease", O_RDONLY);
62 + if (fd < 0)
63 + return -1;
64 +
65 + ssize_t len = read(fd, version, sizeof(version));
66 + if (len < 0)
67 + return -1;
68 +
69 + close(fd);
70 +
71 + char *move = major;
72 + while (*version && *version != '.') *move++ = *version++;
73 + *move = '\0';
74 +
75 + version++;
76 + move = minor;
77 + while (*version && *version != '.') *move++ = *version++;
78 + *move = '\0';
79 +
80 + if (*version)
81 + version++;
82 + move = patch;
83 + while (*version) *move++ = *version++;
84 + *move = '\0';
85 +
86 + return ((int)(str2l(major)*65536) + (int)(str2l(minor)*256) + (int)str2l(patch));
87 +}
88 +
89 +static int has_ebpf_kernel_version(int version) {
90 + return (version >= 264960);
91 +}
92 +
93 +int has_condition_to_run(int version) {
94 + if(!has_ebpf_kernel_version(version))
95 + return 0;
96 +
97 + return 1;
98 +}
libnetdata/ebpf/ebpf.h new
+16
@@ -0,0 +1,16 @@
1 +#ifndef _NETDATA_EBPF_H_
2 +# define _NETDATA_EBPF_H_ 1
3 +
4 +# define NETDATA_DEBUGFS "/sys/kernel/debug/tracing/"
5 +
6 +typedef struct netdata_ebpf_events {
7 + char type;
8 + char *name;
9 +
10 +} netdata_ebpf_events_t;
11 +
12 +extern int clean_kprobe_events(FILE *out, int pid, netdata_ebpf_events_t *ptr);
13 +extern int get_kernel_version();
14 +extern int has_condition_to_run(int version);
15 +
16 +#endif
libnetdata/libnetdata.h
+1
@@ -312,6 +312,7 @@ extern char *netdata_configured_host_prefix;
312 #include "log/log.h"
313 #include "procfile/procfile.h"
314 #include "dictionary/dictionary.h"
315 +#include "ebpf/ebpf.h"
316 #include "eval/eval.h"
317 #include "statistical/statistical.h"
318 #include "adaptive_resortable_list/adaptive_resortable_list.h"
netdata-installer.sh
+5 -1
@@ -856,6 +856,11 @@ if [ "${UID}" -eq 0 ]; then
856 run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/ioping"
857 fi
858
859 + if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/ebpf_process.plugin" ]; then
860 + run chown root:${NETDATA_GROUP} "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/ebpf_process.plugin"
861 + run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/ebpf_process.plugin"
862 + fi
863 +
864 if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/cgroup-network" ]; then
865 run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/cgroup-network"
866 run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/cgroup-network"
@@ -865,7 +870,6 @@ if [ "${UID}" -eq 0 ]; then
870 run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/cgroup-network-helper.sh"
871 run chmod 0750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/cgroup-network-helper.sh"
872 fi
868 -
873 else
874 # non-privileged user installation
875 run chown "${NETDATA_USER}:${NETDATA_GROUP}" "${NETDATA_LOG_DIR}"
netdata.spec.in
+4
@@ -273,6 +273,10 @@ install -m 4750 -p apps.plugin "${RPM_BUILD_ROOT}%{_libexecdir}/%{name}/plugins.
273 # Install perf.plugin
274 install -m 4750 -p perf.plugin "${RPM_BUILD_ROOT}%{_libexecdir}/%{name}/plugins.d/perf.plugin"
275
276 +# ###########################################################
277 +# Install ebpf_process.plugin
278 +install -m 4750 -p ebpf_process.plugin "${RPM_BUILD_ROOT}%{_libexecdir}/%{name}/plugins.d/ebpf_process.plugin"
279 +
280 # ###########################################################
281 # Install cups.plugin
282 %if 0%{?centos_ver} != 6 && 0%{?centos_ver} != 7
web/gui/dashboard_info.js
+49 -1
@@ -540,8 +540,13 @@ netdataDashboard.menu = {
540 title: 'CockroachDB',
541 icon: '<i class="fas fa-database"></i>',
542 info: 'Performance and health statistics for various <code>CockroachDB</code> components.'
543 - }
543 + },
544
545 + 'ebpf': {
546 + title: 'eBPF',
547 + icon: '<i class="fas fa-heartbeat"></i>',
548 + info: 'Monitor system calls, internal functtions, bytes read, bytes written and errors using <code>eBPF</code>.'
549 + }
550 };
551
552
@@ -3012,4 +3017,47 @@ netdataDashboard.context = {
3017 info: 'Size of metric samples written to disk.'
3018 },
3019
3020 + // ------------------------------------------------------------------------
3021 + // eBPF
3022 +
3023 + 'ebpf.file_descriptor': {
3024 + info: 'File descriptor shows the number of calls for internal functions on Linux kernel. The open dimension is attached to the kernel internal function \'do_sys_open\', that is the common function called from open(2) and openat(2). The close dimension is attached to the function \'__close_fd\', that is called from system call close(2).'
3025 + },
3026 +
3027 + 'ebpf.file_error': {
3028 + info: 'File error shows the number of calls that returned an error when called per period.'
3029 + },
3030 +
3031 + 'ebpf.deleted_objects': {
3032 + info: 'Deleted objects monitors calls to the function \'vfs_unlink\'. This chart does not show all events to remove files from the file system, because file systems can create their own functions to remove files.'
3033 + },
3034 +
3035 + 'ebpf.io': {
3036 + info: 'IO shows the number of calls for functions \'vfs_read\' and \'vfs_write\' independent of the return to be success or fail. Like the chart \'deleted_objects\', case the file system uses other function to store data on disks, this chart will not show events for it.'
3037 + },
3038 +
3039 + 'ebpf.io_bytes': {
3040 + info: 'IO bytes shows the total of bytes read or written with success using the functions \'vfs_read\' and \'vfs_write\'.'
3041 + },
3042 +
3043 + 'ebpf.io_error': {
3044 + info: 'IO error shows the number of calls for \'vfs_read\' and \'vfs_write\' that did not have success.'
3045 + },
3046 +
3047 + 'ebpf.process_thread': {
3048 + info: 'Process thread counts the number of times that the function \'do_fork\' was called to create a new task. Task is the common name used to define process and tasks inside the kernel, to identify the threads, Netdata also counts the number of calls for \'sys_clone\' that has the flag \'CLONE_THREAD\' set.'
3049 + },
3050 +
3051 + 'ebpf.exit': {
3052 + info: 'Exit count the number of calls for the functions responsible to close (\'do_exit\') and release(\'release_task\') tasks.'
3053 + },
3054 +
3055 + 'ebpf.task_error': {
3056 + info: 'Task error count the number of errors to create a new process or thread.'
3057 + },
3058 +
3059 + 'ebpf.process_status': {
3060 + info: 'This chart demonstrate the difference between the number of process created and the number of threads created per period(\'process\' dimension), it also shows the number of possible zombie process running on system.'
3061 + }
3062 +
3063 };