e22ec8067 · cryptotaxi247/netdata-1

@@ -1,141 +1,67 @@

1 <!--

2 ----

2 title: "eBPF monitoring with Netdata"

3 +description: "Use Netdata's extended Berkeley Packet Filter (eBPF) collector to monitor kernel-level metrics about your complex applications with per-second granularity."

4 custom_edit_url: https://github.com/netdata/netdata/edit/master/collectors/ebpf.plugin/README.md

5 ----

5 +sidebar_label: "eBPF"

6 -->

7

8 # eBPF monitoring with Netdata

9

10 -This collector plugin uses eBPF (Extended Berkeley Packet Filter) to monitor system calls inside your operating system's

11 -kernel. For now, the main goal of this plugin is to monitor IO and process management on the host where it is running.

10 +Netdata's extended Berkeley Packet Filter (eBPF) collector monitors kernel-level metrics for file descriptors, virtual

11 +filesystem IO, and process management on Linux systems. You can use our eBPF collector to analyze how and when a process

12 +accesses files, when it makes system calls, whether it leaks memory or creating zombie processes, and more.

13 +

14 +Netdata's eBPF monitoring toolkit uses two custom eBPF programs. The default, called `entry`, monitors calls to a

15 +variety of kernel functions, such as `do_sys_open`, `__close_fd`, `vfs_read`, `vfs_write`, `_do_fork`, and more. The

16 +`return` program also monitors the return of each kernel functions to deliver more granular metrics about how your

17 +system and its applications interact with the Linux kernel.

18 +

19 +We expect eBPF monitoring to be particularly valuable in observing and debugging how the Linux kernel handles custom

20 +applications.

21

22 <figure>

23 <img src="https://user-images.githubusercontent.com/1153921/74746434-ad6a1e00-5222-11ea-858a-a7882617ae02.png" alt="An example of VFS charts, made possible by the eBPF collector plugin" />

15 - <figcaption>An example of VFS charts, made possible by the eBPF collector plugin</figcaption>

24 + <figcaption>An example of VFS charts made possible by the eBPF collector plugin.</figcaption>

25 </figure>

26

18 -With this eBPF collector, you can monitor sophisticated system-level metrics about your complex applications while

19 -maintaining Netdata's [high standards for performance](#performance).

20 -

27 ## Enable the collector on Linux

28

23 -eBPF is only available on Linux systems, which means this collector only works on Linux.

24 -

25 -The collector is currently in an _alpha_ stage, as we are still working on improving compatibility with more Linux

26 -distributions and versions, and to ensure the collector works as expected.

27 -

28 -Follow the next few steps to ensure compatibility, prepare your system, install Netdata with eBPF compiled, and enable

29 -the collector.

30 -

31 -### Ensure kernel compatibility

32 -

33 -To enable this plugin and its collector, you must be on a Linux system with a kernel that is more recent than `4.11.0`

34 -and compiled with the option `CONFIG_KPROBES=y`. You can verify whether your kernel has this option enabled by running

35 -the following commands:

36 -

37 -```bash

38 -grep CONFIG_KPROBES=y /boot/config-$(uname -r)

39 -zgrep CONFIG_KPROBES=y /proc/config.gz

40 -```

41 -

42 -If `Kprobes` is enabled, you will see `CONFIG_KPROBES=y` as the command's output, and can skip ahead to the next step: [mount `debugfs` and `tracefs`](#mount-debugfs-and-tracefs).

43 -

44 -If you don't see `CONFIG_KPROBES=y` for any of the commands above, you will have to recompile your kernel to enable it.

45 -

46 -The process of recompiling Linux kernels varies based on your distribution and version. Read the documentation for your

47 -system's distribution to learn more about the specific workflow for recompiling the kernel, ensuring that you set the

48 -`CONFIG_KPROBES` setting to `y` in the process.

49 -

50 -- [Ubuntu](https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel)

51 -- [Debian](https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-common-official)

52 -- [Fedora](https://fedoraproject.org/wiki/Building_a_custom_kernel)

53 -- [CentOS](https://wiki.centos.org/HowTos/Custom_Kernel)

54 -- [Arch Linux](https://wiki.archlinux.org/index.php/Kernel/Traditional_compilation)

55 -- [Slackware](https://docs.slackware.com/howtos:slackware_admin:kernelbuilding)

56 -

57 -### Mount `debugfs` and `tracefs`

58 -

59 -The eBPF collector also requires both the `tracefs` and `debugfs` filesystems. Try mounting the `tracefs` and `debugfs`

60 -filesystems using the commands below:

61 -

62 -```bash

63 -sudo mount -t debugfs nodev /sys/kernel/debug

64 -sudo mount -t tracefs nodev /sys/kernel/tracing

65 -```

66 -

67 -If they are already mounted, you will see an error. If they are not mounted, they should be after running those two

68 -commands. You can also configure your system's `/etc/fstab` configuration to mount these filesystems.

69 -

70 -### Install Netdata with the `--enable-ebpf`

71 -

72 -eBPF collection is only enabled if you install Netdata with the `--enable-ebpf` option.

73 -

74 -If you installed via the [one-line installation script](/packaging/installer/README.md), [64-bit

75 -binary](/packaging/installer/methods/kickstart-64.md), or [manually](/packaging/installer/methods/manual.md), you can

76 -append the `--enable-ebpf` option when you reinstall.

77 -

78 -For example, if you used the one-line installation script, you can reinstall Netdata with the following:

79 -

80 -```bash

81 -bash <(curl -Ss https://my-netdata.io/kickstart.sh) --enable-ebpf

82 -```

83 -

84 -This process will not overwrite any changes you made to configuration files.

85 -

86 -### Edit `netdata.conf` to enable the collector

87 -

88 -After installing Netdata with the `--enable-ebpf` option, you still need to enable the plugin explicitly. To do so, use

89 -`edit-config` to open `netdata.conf` and set `ebpf = yes` in the `[plugins]` section.

90 -

91 -```bash

92 -cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/

93 -./edit-config netdata.conf

94 -```

95 -

96 -Scroll down to the `[plugins]` section and uncomment the `ebpf` line after changing its setting to `yes`.

97 -

98 -```conf

99 -[plugins]

100 - ebpf = yes

101 -```

29 +**The eBPF collector is installed and enabled by default on new nightly installations of the Agent**. eBPF monitoring

30 +only works on Linux systems and with specific Linux kernels, including all kernels newer than `4.11.0`, and all kernels

31 +on CentOS 7.6 or later.

32

103 -Restart Netdata with `service netdata restart`, or the appropriate method for your system, and reload your browser to

104 -see eBPF charts.

33 +If your Agent is v1.22 or older, you may to enable the collector yourself. See the [configuration](#configuration)

34 +section for details.

35

36 ## Charts

37

108 -The first version of `ebpf.plugin` gives a general vision about process running on computer. The charts related

109 -to this plugin are inside the **eBPF** option on dashboard menu and divided in three groups `file`, `vfs`, and

110 -`process`.

111 -

112 -All the collector charts show values per second. The collector retains the total value, but charts only show the

113 -difference between the previous and current metrics collections.

38 +The eBPF collector creates an **eBPF** menu in the Agent's dashboard along with three sub-menus: **File**, **VFS**, and

39 +**Process**. All the charts in this section update every second. The collector stores the actual value inside of its

40 +process, but charts only show the difference between the values collected in the previous and current seconds.

41

42 ### File

43

117 -This group has two charts to demonstrate how software interacts with the Linux kernel to open and close file

118 -descriptors.

44 +This group has two charts demonstrating how software interacts with the Linux kernel to open and close file descriptors.

45

46 #### File descriptor

47

122 -This chart contains two dimensions that show the number of calls to the functions `do_sys_open` and `__close_fd`. These

123 -functions are not commonly called from software, but they are behind the system cals `open(2)`, `openat(2)`, and

124 -`close(2)`.

48 +This chart contains two dimensions that show the number of calls to the functions `do_sys_open` and `__close_fd`. Most

49 +software do not commonly call these functions directly, but they are behind the system calls `open(2)`, `openat(2)`,

50 +and `close(2)`.

51

52 #### File error

53

54 This charts demonstrate the number of times some software tried and failed to open or close a file descriptor.

129 -

55 +

56 ### VFS

57

58 A [virtual file system](https://en.wikipedia.org/wiki/Virtual_file_system) (VFS) is a layer on top of regular

59 filesystems. The functions present inside this API are used for all filesystems, so it's possible the charts in this

134 -group won't show _all_ the actions that occured on your system.

60 +group won't show _all_ the actions that occurred on your system.

61

62 #### Deleted objects

63

138 -This chart monitors calls for `vfs_unlink`. This function is responsible for removing object from the file system.

64 +This chart monitors calls for `vfs_unlink`. This function is responsible for removing objects from the file system.

65

66 #### IO

67

@@ -145,30 +71,30 @@ This chart shows the number of calls to the functions `vfs_read` and `vfs_write`

71

72 This chart also monitors `vfs_read` and `vfs_write`, but instead shows the total of bytes read and written with these

73 functions.

148 -

149 -Netdata displays the number of bytes written as negative, because they are moving down to disk.

150 -

74 +

75 +The Agent displays the number of bytes written as negative because they are moving down to disk.

76 +

77 #### IO errors

78

153 -Netdata counts and shows the number of instances where a running program experiences a read or write error.

79 +The Agent counts and shows the number of instances where a running program experiences a read or write error.

80

81 ### Process

82

83 For this group, the eBPF collector monitors process/thread creation and process end, and then displays any errors in the

84 following charts.

159 -

85 +

86 #### Process thread

87

162 -Internally, the Linux kernel treats both process and threads as `tasks`. To create a thread, the kernel offers a few

163 -system calls: `fork(2)`, `vfork(2)` and `clone(2)`. Each of these system calls in turn use the function `_do_fork`. To

164 -generate this chart, Netdata monitors `_do_fork` to populate the `process` dimension, and monitors `sys_clone` to

165 -identify threads

88 +Internally, the Linux kernel treats both processes and threads as `tasks`. To create a thread, the kernel offers a few

89 +system calls: `fork(2)`, `vfork(2)` and `clone(2)`. In turn, each of these system calls use the function `_do_fork`. To

90 +generate this chart, the eBPF collector monitors `_do_fork` to populate the `process` dimension, and monitors

91 +`sys_clone` to identify threads.

92

93 #### Exit

94

169 -Ending a task is actually two steps. The first is a call to the internal function `do_exit`, which notifies the

170 -operating system that the task is finishing its work. The second step is the release of kernel information, which is

171 -done with the internal function `release_task`. The difference between the two dimensions can help you discover [zombie

95 +Ending a task requires two steps. The first is a call to the internal function `do_exit`, which notifies the operating

96 +system that the task is finishing its work. The second step is to release the kernel information with the internal

97 +function `release_task`. The difference between the two dimensions can help you discover [zombie

98 processes](https://en.wikipedia.org/wiki/Zombie_process).

99

100 #### Task error

@@ -178,20 +104,31 @@ process and thread creation.

104

105 ## Configuration

106

181 -This plugin has different configuration modes, all of which can be adjusted with its configuration file at

182 -`ebpf.conf`. By default, the plugin uses the less expensive `entry` mode. You can learn more about how the

183 -plugin works using `entry` by reading this configuration file.

107 +Enable or disable the entire eBPF collector by editing `netdata.conf`.

108

185 -You can always edit this file with `edit-config`:

109 +```bash

110 +cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/

111 +./edit-config netdata.conf

112 +```

113 +

114 +To enable the collector, scroll down to the `[plugins]` section ensure the relevant line references `ebpf` (not

115 +`ebpf_process`), is uncommented, and is set to `yes`.

116 +

117 +```conf

118 +[plugins]

119 + ebpf = yes

120 +```

121 +

122 +You can also configure the eBPF collector's behavior by editing `ebpf.conf`.

123

124 ```bash

188 -cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/

125 +cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/

126 ./edit-config ebpf.conf

127 ```

128

129 ### `[global]`

130

194 -In this section we define variables applied to the whole collector and the other subsections.

131 +The `[global]` section defines settings for the whole eBPF collector.

132

133 #### load

134

@@ -199,14 +136,68 @@ The collector has two different eBPF programs. These programs monitor the same f

136 monitor, process, and display different kinds of information.

137

138 By default, this plugin uses the `entry` mode. Changing this mode can create significant overhead on your operating

202 -system, but also offer important information if you are developing or debugging software. The `load` option accepts the

139 +system, but also offer valuable information if you are developing or debugging software. The `load` option accepts the

140 following values:

141

205 -- `entry`: This is the default mode. In this mode, Netdata monitors only calls for the functions described in the

206 - sections above. When this mode is selected, Netdata does not show charts related to errors.

207 -- `return`: In this mode, Netdata also monitors the calls to function. In the `entry` mode, Netdata only traces kernel

208 - functions, but with `return`, Netdata also monitors the return of each function. This mode creates more charts, but

209 - also creates an overhead of roughly 110 nanosections for each function call.

142 +- `entry`: This is the default mode. In this mode, the eBPF collector only monitors calls for the functions described

143 + in the sections above, and does not show charts related to errors.

144 +- `return`: In the `return` mode, the eBPF collector monitors the same kernel functions as `entry`, but also creates

145 + new charts for the return of these functions, such as errors. Monitoring function returns can help in debugging

146 + software, such as failing to close file descriptors or creating zombie processes.

147 +

148 +## Troubleshooting

149 +

150 +If the eBPF collector does not work, you can troubleshoot it by running the `ebpf.plugin` command and investigating its output.

151 +

152 +```bash

153 +cd /usr/libexec/netdata/plugins.d/

154 +sudo -u netdata bash

155 +./ebpf.plugin

156 +```

157 +

158 +You can also use `grep` to search the Agent's `error.log` for messages related to eBPF monitoring.

159 +

160 +```bash

161 +grep -i ebpf /var/log/netdata/error.log

162 +```

163 +

164 +### Confirm kernel compatibility

165 +

166 +The eBPF collector only works on Linux systems and with specific Linux kernels. We support all kernels more recent than

167 +`4.11.0`, and all kernels on CentOS 7.6 or later.

168 +

169 +You can run our helper script to determine whether your system can support eBPF monitoring.

170 +

171 +```bash

172 +curl -sSL https://raw.githubusercontent.com/netdata/kernel-collector/master/tools/check-kernel-config.sh | sudo sh

173 +```

174 +

175 +If this script returns no output, your system is ready to compile and run the eBPF collector.

176 +

177 +If you see a warning about a missing kerkel configuration (`KPROBES KPROBES_ON_FTRACE HAVE_KPROBES BPF BPF_SYSCALL

178 +BPF_JIT`), you will need to recompile your kernel to support this configuration. The process of recompiling Linux

179 +kernels varies based on your distribution and version. Read the documentation for your system's distribution to learn

180 +more about the specific workflow for recompiling the kernel, ensuring that you set all the necessary

181 +

182 +- [Ubuntu](https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel)

183 +- [Debian](https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-common-official)

184 +- [Fedora](https://fedoraproject.org/wiki/Building_a_custom_kernel)

185 +- [CentOS](https://wiki.centos.org/HowTos/Custom_Kernel)

186 +- [Arch Linux](https://wiki.archlinux.org/index.php/Kernel/Traditional_compilation)

187 +- [Slackware](https://docs.slackware.com/howtos:slackware_admin:kernelbuilding)

188 +

189 +### Mount `debugfs` and `tracefs`

190 +

191 +The eBPF collector also requires both the `tracefs` and `debugfs` filesystems. Try mounting the `tracefs` and `debugfs`

192 +filesystems using the commands below:

193 +

194 +```bash

195 +sudo mount -t debugfs nodev /sys/kernel/debug

196 +sudo mount -t tracefs nodev /sys/kernel/tracing

197 +```

198 +

199 +If they are already mounted, you will see an error. You can also configure your system's `/etc/fstab` configuration to

200 +mount these filesystems on startup.

201

202 ## Performance

203

Update eBPF documentation to reflect default enabled status (#9105)