Update eBPF documentation (#11741)
thiagoftsm committed
Nov 3, 2021 at 20:01 UTC
280b3f4b799b90b59df2b44d1d463ec1c988aee1
1 file changed
+166
-63
collectors/ebpf.plugin/README.md
+166
-63
@@ -8,14 +8,13 @@ sidebar_label: "eBPF"
8
9
# eBPF monitoring with Netdata
10
11
-Netdata's extended Berkeley Packet Filter (eBPF) collector monitors kernel-level metrics for file descriptors, virtual
12
-filesystem IO, and process management on Linux systems. You can use our eBPF collector to analyze how and when a process
13
-accesses files, when it makes system calls, whether it leaks memory or creating zombie processes, and more.
11
+eBPF consists of a wide toolchain that ultimately outputs a set of bytecode that will run inside the eBPF virtual
12
+machine (VM) which lives inside the Linux kernel. The program in particular is executed in response to a [tracepoint
13
+or kprobe](#probes-and-tracepoints) activation.
14
15
-Netdata's eBPF monitoring toolkit uses two custom eBPF programs. The default, called `entry`, monitors calls to a
16
-variety of kernel functions, such as `do_sys_open`, `__close_fd`, `vfs_read`, `vfs_write`, `_do_fork`, and more. The
17
-`return` program also monitors the return of each kernel functions to deliver more granular metrics about how your
18
-system and its applications interact with the Linux kernel.
15
+Netdata has written many eBPF programs, which, when compiled and integrated into the Netdata Agent, are able to collect
16
+a wide array of data about the host that would otherwise be impossible. The data eBPF programs can collect is truly unique,
17
+which gives the Netdata Agent access to data that is high value but normally hard to capture.
18
19
eBPF monitoring can help you troubleshoot and debug how applications interact with the Linux kernel. See
20
our [guide on troubleshooting apps with eBPF metrics](/docs/guides/troubleshoot/monitor-debug-applications-ebpf.md) for
@@ -26,11 +25,28 @@ configuration and troubleshooting tips.
25
<figcaption>An example of VFS charts made possible by the eBPF collector plugin.</figcaption>
26
</figure>
27
29
-## Enable the collector on Linux
28
+## Probes and Tracepoints
29
+
30
+The following two features from the Linux kernel are used by Netdata to run eBPF programs:
31
+
32
+- Kprobes and return probes (kretprobe): Probes can insert virtually into any kernel instruction. When eBPF runs in
33
+ `entry` mode, it attaches only `kprobes` for internal functions monitoring calls and some arguments every time a
34
+ function is called. The user can also change configuration to use [`return`](#global) mode, and this will allow users
35
+ to monitor return from these functions and detect possible failures.
36
+- Tracepoints are hooks to call specific functions. Tracepoints are more stable than `kprobes` and are preferred when
37
+ both options are available.
38
+
39
+In each case, wherever a normal kprobe, kretprobe, or tracepoint would have run its hook function, an eBPF program is
40
+run instead, performing various collection logic before letting the kernel continue its normal control flow.
41
+
42
+There are more methods by which eBPF programs can be triggered but which are not currently supported, such as via uprobes
43
+which allow hooking into arbitrary user-space functions in a similar manner to kprobes.
44
+
45
+## Manually enable the collector on Linux
46
47
**The eBPF collector is installed and enabled by default on most new installations of the Agent**. The eBPF collector
32
-does not currently work with [static build installations](/packaging/installer/methods/kickstart-64.md), but improved
33
-support is in active development.
48
+does not currently work with [static build installations](/packaging/installer/methods/kickstart-64.md) for kernels older
49
+than `4.11`, but improved support is in active development.
50
51
eBPF monitoring only works on Linux systems and with specific Linux kernels, including all kernels newer than `4.11.0`,
52
and all kernels on CentOS 7.6 or later.
@@ -40,9 +56,8 @@ section for details.
56
57
## Charts
58
43
-The eBPF collector creates an **eBPF** menu in the Agent's dashboard along with two sub-menus: **Socket**, and
44
-**Process**. The plugin also creates charts on different menus, like System Overview, Memory, Disks, Filesystem, Mount
45
-Points and Applications. All the charts in this section update every second.
59
+The eBPF collector creates charts on different menus, like System Overview, Memory, MD arrays, Disks, Filesystem,
60
+Mount Points, Networking Stack, systemd Services, and Applications.
61
62
The collector stores the actual value inside of its process, but charts only show the difference between the values
63
collected in the previous and current seconds.
@@ -53,10 +68,60 @@ Not all charts within the System Overview menu are enabled by default, because t
68
function call, this number is small for a human perspective, but the functions are called many times creating an impact
69
on host. See the [configuration](#configuration) section for details about how to enable them.
70
71
+#### Processes
72
+
73
+Internally, the Linux kernel treats both processes and threads as `tasks`. To create a thread, the kernel offers a few
74
+system calls: `fork(2)`, `vfork(2)`, and `clone(2)`. To generate this chart, the eBPF
75
+collector uses the following `tracepoints` and `kprobe`:
76
+
77
+- `sched/sched_process_fork`: Tracepoint called after a call for `fork (2)`, `vfork (2)` and `clone (2)`.
78
+- `sched/sched_process_exec`: Tracepoint called after a exec-family syscall.
79
+- `kprobe/kernel_clone`: This is the main [fork](https://elixir.bootlin.com/linux/v5.10/source/kernel/fork.c#L2415)
80
+ routine since kernel `5.10.0` was released.
81
+- `kprobe/_do_fork`: Like `kernel_clone`, but this was the main function between kernels `4.2.0` and `5.9.16`
82
+- `kprobe/do_fork`: This was the main function before kernel `4.2.0`.
83
+
84
+#### Process Exit
85
+
86
+Ending a task requires two steps. The first is a call to the internal function `do_exit`, which notifies the operating
87
+system that the task is finishing its work. The second step is to release the kernel information with the internal
88
+function `release_task`. The difference between the two dimensions can help you discover
89
+[zombie processes](https://en.wikipedia.org/wiki/Zombie_process). To get the metrics, the collector uses:
90
+
91
+- `sched/sched_process_exit`: Tracepoint called after a task exits.
92
+- `kprobe/release_task`: This function is called when a process exits, as the kernel still needs to remove the process
93
+ descriptor.
94
+
95
+#### Task error
96
+
97
+The functions responsible for ending tasks do not return values, so this chart contains information about failures on
98
+process and thread creation only.
99
+
100
+
101
#### Swap
102
103
Inside the swap submenu the eBPF plugin creates the chart `swapcalls`; this chart is displaying when processes are
59
-calling functions `swap_readpage` and `swap_writepage`, which are functions responsible for doing IO in swap memory.
104
+calling functions [`swap_readpage` and `swap_writepage`](https://hzliu123.github.io/linux-kernel/Page%20Cache%20in%20Linux%202.6.pdf ),
105
+which are functions responsible for doing IO in swap memory. To collect the exact moment that an access to swap happens,
106
+the collector attaches `kprobes` for cited functions.
107
+
108
+#### Soft IRQ
109
+
110
+The following `tracepoints` are used to measure time usage for soft IRQs:
111
+
112
+- [irq/softirq_entry](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_softirq_entry): Called
113
+ before softirq handler
114
+- [irq/softirq_exit](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_softirq_exit): Called when
115
+ softirq handler returns.
116
+
117
+#### IPC shared memory
118
+
119
+To monitor shared memory system call counts, the following `kprobes` are used:
120
+
121
+- `shmget`: Runs when [shmget](https://man7.org/linux/man-pages/man2/shmget.2.html) is called.
122
+- `shmat`: Runs when [shmat](https://man7.org/linux/man-pages/man2/shmat.2.html) is called.
123
+- `shmdt`: Runs when [shmdt](https://man7.org/linux/man-pages/man2/shmat.2.html) is called.
124
+- `shmctl`: Runs when [shmctl](https://man7.org/linux/man-pages/man2/shmctl.2.html) is called.
125
126
### Memory
127
@@ -77,9 +142,13 @@ organization:
142
#### Page cache ratio
143
144
The chart `cachestat_ratio` shows how processes are accessing page cache. In a normal scenario, we expect values around
80
-100%, which means that the majority of the work on the machine is processed in memory. To calculate the ratio Netdata
81
-monitors calls for kernel functions `add_to_page_cache_lru`, `mark_page_accessed`, `account_page_dirtied`, and
82
-`mark_buffer_dirty`.
145
+100%, which means that the majority of the work on the machine is processed in memory. To calculate the ratio, Netdata
146
+attaches `kprobes` for kernel functions:
147
+
148
+- `add_to_page_cache_lru`: Page addition.
149
+- `mark_page_accessed`: Access to cache.
150
+- `account_page_dirtied`: Dirty (modified) pages.
151
+- `mark_buffer_dirty`: Writes to page cache.
152
153
#### Dirty pages
154
@@ -98,29 +167,32 @@ result of the difference for calls between functions `add_to_page_cache_lru` and
167
168
#### File sync
169
101
-This chart shows calls to synchronization methods, `fsync(2)` and `fdatasync(2)`, to transfer all modified page caches
170
+This chart shows calls to synchronization methods, [fsync (2)](https://man7.org/linux/man-pages/man2/fdatasync.2.html)
171
+and [fdatasync (2)](https://man7.org/linux/man-pages/man2/fdatasync.2.html), to transfer all modified page caches
172
for the files on disk devices. These calls block until the disk reports that the transfer has been completed. They flush
173
data for specific file descriptors.
174
175
#### Memory map sync
176
107
-The chart shows calls to `msync(2)` syscalls. This syscall flushes changes to a file that was mapped into memory using
108
-`mmap(2)`.
177
+The chart shows calls to [msync (2)](https://man7.org/linux/man-pages/man2/msync.2.html) syscalls. This syscall flushes
178
+changes to a file that was mapped into memory using [mmap (2)](https://man7.org/linux/man-pages/man2/mmap.2.html).
179
180
#### File system sync
181
112
-This chart monitors calls demonstrating commits from filesystem caches to disk.
182
+This chart monitors calls demonstrating commits from filesystem caches to disk. Netdata attaches `kprobes` for
183
+[sync (2)](https://man7.org/linux/man-pages/man2/sync.2.html), and [syncfs (2)](https://man7.org/linux/man-pages/man2/sync.2.html).
184
185
#### File range sync
186
116
-This chart shows calls to `sync_file_range(2)` which synchronizes file segments with disk. This is the most dangerous
117
-syscall to synchronize data according to its manual.
187
+This chart shows calls to [sync_file_range (2)](https://man7.org/linux/man-pages/man2/sync_file_range.2.html) which
188
+synchronizes file segments with disk.
189
119
-### MD flush
190
+> Note: This is the most dangerous syscall to synchronize data, according to its manual.
191
121
-The eBPF plugin shows multi-device flushes happening in real time. This can be
122
-used to explain some spikes happening in
123
-[disk latency](#disk) charts.
192
+### Multiple Device (MD) arrays
193
+
194
+The eBPF plugin shows multi-device flushes happening in real time. This can be used to explain some spikes happening
195
+in [disk latency](#disk) charts.
196
197
By default, MD flush is disabled. To enable it, configure your
198
`/etc/netdata/ebpf.d.conf` file as:
@@ -130,87 +202,112 @@ By default, MD flush is disabled. To enable it, configure your
202
mdflush = yes
203
```
204
205
+#### MD flush
206
+
207
+To collect data related to Linux multi-device (MD) flushing, the following kprobe is used:
208
+
209
+- `kprobe/md_flush_request`: called whenever a request for flushing multi-device data is made.
210
+
211
### Disk
212
213
The eBPF plugin also shows a chart in the Disk section when the `disk` thread is enabled. This will create the
136
-chart `disk_latency_io` for each disk on the host. These charts
137
-use [tracepoints](https://www.kernel.org/doc/html/latest/trace/tracepoints.html)
138
-`block:block_rq_issue` and `block:block_rq_complete` to measure the latency of IO events.
214
+chart `disk_latency_io` for each disk on the host. The following tracepoints are used:
215
+
216
+- [block/block_rq_issue](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_block_rq_issue):
217
+ IO request operation to a device drive.
218
+- [block:block_rq_complete](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_block_rq_complete):
219
+ IO operation completed by device.
220
221
### Filesystem
222
142
-This group has two charts demonstrating how applications interact with the Linux kernel to open and close file
143
-descriptors.
223
+This group has charts demonstrating how applications interact with the Linux kernel to open and close file
224
+descriptors. It also brings latency charts for five different filesystems and monitoring for Linux Virtual File System (VFS),
225
+that is a layer on top of regular filesystems. The functions presented inside this API are not used for filesystems, so
226
+it's possible that the charts in this section won't show _all_ the actions that occurred on your system.
227
+
228
+#### btrfs
229
+
230
+To measure the latency of executing some actions in a [btrfs](https://elixir.bootlin.com/linux/latest/source/fs/btrfs/file.c)
231
+filesystem, the collector needs to attach `kprobes` and `kretprobes` for each one of the following functions:
232
+
233
+> Note: We are listing two functions used to measure `read` latency, but we use either `btrfs_file_read_iter` or
234
+`generic_file_read_iter`, depending on kernel version.
235
+
236
+- `btrfs_file_read_iter`: Function used to measure read latency since kernel `5.10.0`.
237
+- `generic_file_read_iter`: Like `btrfs_file_read_iter`, but this function was used before kernel `5.10.0`.
238
+- `btrfs_file_write_iter`: Function used to write data.
239
+- `btrfs_file_open`: Function used to open files.
240
+- `btrfs_sync_file`: Function used to synchronize data to filesystem.
241
242
#### File descriptor
243
147
-This chart contains two dimensions that show the number of calls to the functions `do_sys_open` and `__close_fd`. Most
148
-software doesn't commonly call these functions directly, they are behind the system calls `open(2)`, `openat(2)`,
149
-and `close(2)`.
244
+To give metrics related to `open` and `close` events, instead of attaching kprobes for each syscall used to do these
245
+events, the collector attaches `kprobes` for the common function used for syscalls:
246
+
247
+- [do_sys_open](https://0xax.gitbooks.io/linux-insides/content/SysCall/linux-syscall-5.html ): Internal function used to
248
+ open files.
249
+- [do_sys_openat2](https://elixir.bootlin.com/linux/v5.6/source/fs/open.c#L1162): Function called from do_sys_open since
250
+ version `5.6.0`.
251
+- [close_fd](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg2271761.html): Function used to close file
252
+ descriptor since kernel `5.11.0`.
253
+- `__close_fd`: Function used to close files before version `5.11.0`.
254
255
#### File error
256
257
This chart shows the number of times some software tried and failed to open or close a file descriptor.
258
155
-### VFS
156
-
157
-A [virtual file system](https://en.wikipedia.org/wiki/Virtual_file_system) (VFS) is a layer on top of regular
158
-filesystems. The functions presented inside this API are not used for filesystems, so it's possible that the charts in
159
-this section won't show _all_ the actions that occurred on your system.
160
-
161
-#### Deleted objects
259
+#### VFS Deleted objects
260
261
This chart monitors calls to `vfs_unlink`. This function is responsible for removing objects from the file system.
262
165
-#### IO
263
+#### VFS IO
264
265
This chart shows the number of calls to the functions `vfs_read` and `vfs_write`.
266
169
-#### IO bytes
267
+#### VFS IO bytes
268
269
This chart also monitors `vfs_read` and `vfs_write` but, instead of the number of calls, it shows the total amount of
270
bytes read and written with these functions.
271
272
The Agent displays the number of bytes written as negative because they are moving down to disk.
273
176
-#### IO errors
274
+#### VFS IO errors
275
276
The Agent counts and shows the number of instances where a running program experiences a read or write error.
277
180
-#### Create
278
+#### VFS Create
279
280
This chart shows the number of calls to `vfs_create`. This function is responsible for creating files.
281
184
-#### Synchronization
282
+#### VFS Synchronization
283
284
This chart shows the number of calls to `vfs_fsync`. This function is responsible for calling `fsync(2)` or
285
`fdatasync(2)` on a file. You can see more details in the Synchronization section.
286
189
-#### Open
287
+#### VFS Open
288
289
This chart shows the number of calls to `vfs_open`. This function is responsible for opening files.
290
193
-### eBPF
291
+#### Directory Cache
292
195
-#### Process thread
293
+Metrics for directory cache are collected using kprobe for `lookup_fast`, because we are interested in the number of
294
+times this function is accessed. On the other hand, for `d_lookup` we are not only interested in the number of times it
295
+is accessed, but also in possible errors, so we need to attach a `kretprobe`. For this reason, the following is used:
296
197
-Internally, the Linux kernel treats both processes and threads as `tasks`. To create a thread, the kernel offers a few
198
-system calls: `fork(2)`, `vfork(2)`, and `clone(2)`. In turn, each of these system calls uses either the function
199
-`_do_fork` (kernel older than `5.10.0`) or the function `do_fork` (latest kernels). To generate this chart, the eBPF
200
-collector monitors the cited functions to populate the `process` dimension, and monitors `sys_clone` to identify
201
-threads.
297
+- [lookup_fast](https://lwn.net/Articles/649115/): Called to look at data inside the directory cache.
298
+- [d_lookup](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/dcache.c?id=052b398a43a7de8c68c13e7fa05d6b3d16ce6801#n2223):
299
+ Called when the desired file is not inside the directory cache.
300
203
-#### Exit
301
+### Mount Points
302
205
-Ending a task requires two steps. The first is a call to the internal function `do_exit`, which notifies the operating
206
-system that the task is finishing its work. The second step is to release the kernel information with the internal
207
-function `release_task`. The difference between the two dimensions can help you discover
208
-[zombie processes](https://en.wikipedia.org/wiki/Zombie_process).
303
+The following `kprobes` are used to collect `mount` & `unmount` call counts:
304
210
-#### Task error
305
+- [mount](https://man7.org/linux/man-pages/man2/mount.2.html): mount filesystem on host.
306
+- [umount](https://man7.org/linux/man-pages/man2/umount.2.html): umount filesystem on host.
307
212
-The functions responsible for ending tasks do not return values, so this chart contains information about failures on
213
-process and thread creation only.
308
+### Networking Stack
309
+
310
+Netdata monitors socket bandwidth attaching `kprobes` for internal functions.
311
312
#### TCP functions
313
@@ -237,6 +334,12 @@ receiving data for connections when the `UDP` protocol is used.
334
Like the previous chart, this one also monitors `udp_sendmsg` and `udp_recvmsg`, but instead of showing the number of
335
calls, it monitors the number of bytes sent and received.
336
337
+### Apps
338
+
339
+These are tracepoints related to [OOM](https://en.wikipedia.org/wiki/Out_of_memory) killing processes.
340
+
341
+- `oom/mark_victm`: Monitors when an oomkill event happens.
342
+
343
## Configuration
344
345
Enable or disable the entire eBPF collector by editing `netdata.conf`.