@cryptotaxi247 / netdata-1 / commits / fd819dd69

Update eBPF documenation (Filesystem and HardIRQ) (#11752)

* ebpf docs: remove trailing spaces and extra newlines. * ebpf docs: add backticks on funcs/tracepoints/kprobe names. * ebpf docs: typo fix. * ebpf docs: add hard irq. * ebpf docs: add docs for remaining filesystems. * ebpf docs: address review. * ebpf docs: address review.

Uman Shahzad committed Nov 9, 2021 at 11:20 UTC fd819dd69f123f5d94fde157265da4ccd41e8e33
1 file changed +174 -60
collectors/ebpf.plugin/README.md
+174 -60
@@ -1,6 +1,6 @@
1 <!--
2 title: "eBPF monitoring with Netdata"
3 -description: "Use Netdata's extended Berkeley Packet Filter (eBPF) collector to monitor kernel-level metrics about your
3 +description: "Use Netdata's extended Berkeley Packet Filter (eBPF) collector to monitor kernel-level metrics about your
4 complex applications with per-second granularity."
5 custom_edit_url: https://github.com/netdata/netdata/edit/master/collectors/ebpf.plugin/README.md
6 sidebar_label: "eBPF"
@@ -8,12 +8,12 @@ sidebar_label: "eBPF"
8
9 # eBPF monitoring with Netdata
10
11 -eBPF consists of a wide toolchain that ultimately outputs a set of bytecode that will run inside the eBPF virtual
12 -machine (VM) which lives inside the Linux kernel. The program in particular is executed in response to a [tracepoint
11 +eBPF consists of a wide toolchain that ultimately outputs a set of bytecode that will run inside the eBPF virtual
12 +machine (VM) which lives inside the Linux kernel. The program in particular is executed in response to a [tracepoint
13 or kprobe](#probes-and-tracepoints) activation.
14
15 -Netdata has written many eBPF programs, which, when compiled and integrated into the Netdata Agent, are able to collect
16 -a wide array of data about the host that would otherwise be impossible. The data eBPF programs can collect is truly unique,
15 +Netdata has written many eBPF programs, which, when compiled and integrated into the Netdata Agent, are able to collect
16 +a wide array of data about the host that would otherwise be impossible. The data eBPF programs can collect is truly unique,
17 which gives the Netdata Agent access to data that is high value but normally hard to capture.
18
19 eBPF monitoring can help you troubleshoot and debug how applications interact with the Linux kernel. See
@@ -29,17 +29,17 @@ configuration and troubleshooting tips.
29
30 The following two features from the Linux kernel are used by Netdata to run eBPF programs:
31
32 -- Kprobes and return probes (kretprobe): Probes can insert virtually into any kernel instruction. When eBPF runs in
33 - `entry` mode, it attaches only `kprobes` for internal functions monitoring calls and some arguments every time a
34 - function is called. The user can also change configuration to use [`return`](#global) mode, and this will allow users
32 +- Kprobes and return probes (kretprobe): Probes can insert virtually into any kernel instruction. When eBPF runs in
33 + `entry` mode, it attaches only `kprobes` for internal functions monitoring calls and some arguments every time a
34 + function is called. The user can also change configuration to use [`return`](#global) mode, and this will allow users
35 to monitor return from these functions and detect possible failures.
36 -- Tracepoints are hooks to call specific functions. Tracepoints are more stable than `kprobes` and are preferred when
36 +- Tracepoints are hooks to call specific functions. Tracepoints are more stable than `kprobes` and are preferred when
37 both options are available.
38
39 -In each case, wherever a normal kprobe, kretprobe, or tracepoint would have run its hook function, an eBPF program is
39 +In each case, wherever a normal kprobe, kretprobe, or tracepoint would have run its hook function, an eBPF program is
40 run instead, performing various collection logic before letting the kernel continue its normal control flow.
41
42 -There are more methods by which eBPF programs can be triggered but which are not currently supported, such as via uprobes
42 +There are more methods by which eBPF programs can be triggered but which are not currently supported, such as via uprobes
43 which allow hooking into arbitrary user-space functions in a similar manner to kprobes.
44
45 ## Manually enable the collector on Linux
@@ -76,7 +76,7 @@ collector uses the following `tracepoints` and `kprobe`:
76
77 - `sched/sched_process_fork`: Tracepoint called after a call for `fork (2)`, `vfork (2)` and `clone (2)`.
78 - `sched/sched_process_exec`: Tracepoint called after a exec-family syscall.
79 -- `kprobe/kernel_clone`: This is the main [fork](https://elixir.bootlin.com/linux/v5.10/source/kernel/fork.c#L2415)
79 +- `kprobe/kernel_clone`: This is the main [`fork()`](https://elixir.bootlin.com/linux/v5.10/source/kernel/fork.c#L2415)
80 routine since kernel `5.10.0` was released.
81 - `kprobe/_do_fork`: Like `kernel_clone`, but this was the main function between kernels `4.2.0` and `5.9.16`
82 - `kprobe/do_fork`: This was the main function before kernel `4.2.0`.
@@ -97,31 +97,65 @@ function `release_task`. The difference between the two dimensions can help you
97 The functions responsible for ending tasks do not return values, so this chart contains information about failures on
98 process and thread creation only.
99
100 -
100 #### Swap
101
102 Inside the swap submenu the eBPF plugin creates the chart `swapcalls`; this chart is displaying when processes are
104 -calling functions [`swap_readpage` and `swap_writepage`](https://hzliu123.github.io/linux-kernel/Page%20Cache%20in%20Linux%202.6.pdf ),
105 -which are functions responsible for doing IO in swap memory. To collect the exact moment that an access to swap happens,
103 +calling functions [`swap_readpage` and `swap_writepage`](https://hzliu123.github.io/linux-kernel/Page%20Cache%20in%20Linux%202.6.pdf ),
104 +which are functions responsible for doing IO in swap memory. To collect the exact moment that an access to swap happens,
105 the collector attaches `kprobes` for cited functions.
106
107 #### Soft IRQ
108
109 The following `tracepoints` are used to measure time usage for soft IRQs:
110
112 -- [irq/softirq_entry](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_softirq_entry): Called
111 +- [`irq/softirq_entry`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_softirq_entry): Called
112 before softirq handler
114 -- [irq/softirq_exit](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_softirq_exit): Called when
113 +- [`irq/softirq_exit`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_softirq_exit): Called when
114 softirq handler returns.
116 -
115 +
116 +#### Hard IRQ
117 +
118 +The following tracepoints are used to measure the latency of servicing a
119 +hardware interrupt request (hard IRQ).
120 +
121 +- [`irq/irq_handler_entry`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_irq_handler_entry):
122 + Called immediately before the IRQ action handler.
123 +- [`irq/irq_handler_exit`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_irq_handler_exit):
124 + Called immediately after the IRQ action handler returns.
125 +- `irq_vectors`: These are traces from `irq_handler_entry` and
126 + `irq_handler_exit` when an IRQ is handled. The following elements from vector
127 + are triggered:
128 + - `irq_vectors/local_timer_entry`
129 + - `irq_vectors/local_timer_exit`
130 + - `irq_vectors/reschedule_entry`
131 + - `irq_vectors/reschedule_exit`
132 + - `irq_vectors/call_function_entry`
133 + - `irq_vectors/call_function_exit`
134 + - `irq_vectors/call_function_single_entry`
135 + - `irq_vectors/call_function_single_xit`
136 + - `irq_vectors/irq_work_entry`
137 + - `irq_vectors/irq_work_exit`
138 + - `irq_vectors/error_apic_entry`
139 + - `irq_vectors/error_apic_exit`
140 + - `irq_vectors/thermal_apic_entry`
141 + - `irq_vectors/thermal_apic_exit`
142 + - `irq_vectors/threshold_apic_entry`
143 + - `irq_vectors/threshold_apic_exit`
144 + - `irq_vectors/deferred_error_entry`
145 + - `irq_vectors/deferred_error_exit`
146 + - `irq_vectors/spurious_apic_entry`
147 + - `irq_vectors/spurious_apic_exit`
148 + - `irq_vectors/x86_platform_ipi_entry`
149 + - `irq_vectors/x86_platform_ipi_exit`
150 +
151 #### IPC shared memory
152
153 To monitor shared memory system call counts, the following `kprobes` are used:
154
121 -- `shmget`: Runs when [shmget](https://man7.org/linux/man-pages/man2/shmget.2.html) is called.
122 -- `shmat`: Runs when [shmat](https://man7.org/linux/man-pages/man2/shmat.2.html) is called.
123 -- `shmdt`: Runs when [shmdt](https://man7.org/linux/man-pages/man2/shmat.2.html) is called.
124 -- `shmctl`: Runs when [shmctl](https://man7.org/linux/man-pages/man2/shmctl.2.html) is called.
155 +- `shmget`: Runs when [`shmget`](https://man7.org/linux/man-pages/man2/shmget.2.html) is called.
156 +- `shmat`: Runs when [`shmat`](https://man7.org/linux/man-pages/man2/shmat.2.html) is called.
157 +- `shmdt`: Runs when [`shmdt`](https://man7.org/linux/man-pages/man2/shmat.2.html) is called.
158 +- `shmctl`: Runs when [`shmctl`](https://man7.org/linux/man-pages/man2/shmctl.2.html) is called.
159
160 ### Memory
161
@@ -143,7 +177,7 @@ organization:
177
178 The chart `cachestat_ratio` shows how processes are accessing page cache. In a normal scenario, we expect values around
179 100%, which means that the majority of the work on the machine is processed in memory. To calculate the ratio, Netdata
146 -attaches `kprobes` for kernel functions:
180 +attaches `kprobes` for kernel functions:
181
182 - `add_to_page_cache_lru`: Page addition.
183 - `mark_page_accessed`: Access to cache.
@@ -167,31 +201,31 @@ result of the difference for calls between functions `add_to_page_cache_lru` and
201
202 #### File sync
203
170 -This chart shows calls to synchronization methods, [fsync (2)](https://man7.org/linux/man-pages/man2/fdatasync.2.html)
171 -and [fdatasync (2)](https://man7.org/linux/man-pages/man2/fdatasync.2.html), to transfer all modified page caches
204 +This chart shows calls to synchronization methods, [`fsync(2)`](https://man7.org/linux/man-pages/man2/fdatasync.2.html)
205 +and [`fdatasync(2)`](https://man7.org/linux/man-pages/man2/fdatasync.2.html), to transfer all modified page caches
206 for the files on disk devices. These calls block until the disk reports that the transfer has been completed. They flush
207 data for specific file descriptors.
208
209 #### Memory map sync
210
177 -The chart shows calls to [msync (2)](https://man7.org/linux/man-pages/man2/msync.2.html) syscalls. This syscall flushes
178 -changes to a file that was mapped into memory using [mmap (2)](https://man7.org/linux/man-pages/man2/mmap.2.html).
211 +The chart shows calls to [`msync(2)`](https://man7.org/linux/man-pages/man2/msync.2.html) syscalls. This syscall flushes
212 +changes to a file that was mapped into memory using [`mmap(2)`](https://man7.org/linux/man-pages/man2/mmap.2.html).
213
214 #### File system sync
215
182 -This chart monitors calls demonstrating commits from filesystem caches to disk. Netdata attaches `kprobes` for
183 -[sync (2)](https://man7.org/linux/man-pages/man2/sync.2.html), and [syncfs (2)](https://man7.org/linux/man-pages/man2/sync.2.html).
216 +This chart monitors calls demonstrating commits from filesystem caches to disk. Netdata attaches `kprobes` for
217 +[`sync(2)`](https://man7.org/linux/man-pages/man2/sync.2.html), and [`syncfs(2)`](https://man7.org/linux/man-pages/man2/sync.2.html).
218
219 #### File range sync
220
187 -This chart shows calls to [sync_file_range (2)](https://man7.org/linux/man-pages/man2/sync_file_range.2.html) which
221 +This chart shows calls to [`sync_file_range(2)`](https://man7.org/linux/man-pages/man2/sync_file_range.2.html) which
222 synchronizes file segments with disk.
223
224 > Note: This is the most dangerous syscall to synchronize data, according to its manual.
225
226 ### Multiple Device (MD) arrays
227
194 -The eBPF plugin shows multi-device flushes happening in real time. This can be used to explain some spikes happening
228 +The eBPF plugin shows multi-device flushes happening in real time. This can be used to explain some spikes happening
229 in [disk latency](#disk) charts.
230
231 By default, MD flush is disabled. To enable it, configure your
@@ -213,22 +247,69 @@ To collect data related to Linux multi-device (MD) flushing, the following kprob
247 The eBPF plugin also shows a chart in the Disk section when the `disk` thread is enabled. This will create the
248 chart `disk_latency_io` for each disk on the host. The following tracepoints are used:
249
216 -- [block/block_rq_issue](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_block_rq_issue):
250 +- [`block/block_rq_issue`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_block_rq_issue):
251 IO request operation to a device drive.
218 -- [block:block_rq_complete](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_block_rq_complete):
252 +- [`block/block_rq_complete`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_block_rq_complete):
253 IO operation completed by device.
254
255 ### Filesystem
256
223 -This group has charts demonstrating how applications interact with the Linux kernel to open and close file
224 -descriptors. It also brings latency charts for five different filesystems and monitoring for Linux Virtual File System (VFS),
225 -that is a layer on top of regular filesystems. The functions presented inside this API are not used for filesystems, so
226 -it's possible that the charts in this section won't show _all_ the actions that occurred on your system.
257 +This group has charts demonstrating how applications interact with the Linux
258 +kernel to open and close file descriptors. It also brings latency charts for
259 +several different filesystems.
260 +
261 +#### ext4
262 +
263 +To measure the latency of executing some actions in an
264 +[ext4](https://elixir.bootlin.com/linux/latest/source/fs/ext4) filesystem, the
265 +collector needs to attach `kprobes` and `kretprobes` for each of the following
266 +functions:
267 +
268 +- `ext4_file_read_iter`: Function used to measure read latency.
269 +- `ext4_file_write_iter`: Function used to measure write latency.
270 +- `ext4_file_open`: Function used to measure open latency.
271 +- `ext4_sync_file`: Function used to measure sync latency.
272 +
273 +#### ZFS
274 +
275 +To measure the latency of executing some actions in a zfs filesystem, the
276 +collector needs to attach `kprobes` and `kretprobes` for each of the following
277 +functions:
278 +
279 +- `zpl_iter_read`: Function used to measure read latency.
280 +- `zpl_iter_write`: Function used to measure write latency.
281 +- `zpl_open`: Function used to measure open latency.
282 +- `zpl_fsync`: Function used to measure sync latency.
283 +
284 +#### XFS
285 +
286 +To measure the latency of executing some actions in an
287 +[xfs](https://elixir.bootlin.com/linux/latest/source/fs/xfs) filesystem, the
288 +collector needs to attach `kprobes` and `kretprobes` for each of the following
289 +functions:
290 +
291 +- `xfs_file_read_iter`: Function used to measure read latency.
292 +- `xfs_file_write_iter`: Function used to measure write latency.
293 +- `xfs_file_open`: Function used to measure open latency.
294 +- `xfs_file_fsync`: Function used to measure sync latency.
295 +
296 +#### NFS
297 +
298 +To measure the latency of executing some actions in an
299 +[nfs](https://elixir.bootlin.com/linux/latest/source/fs/nfs) filesystem, the
300 +collector needs to attach `kprobes` and `kretprobes` for each of the following
301 +functions:
302 +
303 +- `nfs_file_read`: Function used to measure read latency.
304 +- `nfs_file_write`: Function used to measure write latency.
305 +- `nfs_file_open`: Functions used to measure open latency.
306 +- `nfs4_file_open`: Functions used to measure open latency for NFS v4.
307 +- `nfs_getattr`: Function used to measure sync latency.
308
309 #### btrfs
310
230 -To measure the latency of executing some actions in a [btrfs](https://elixir.bootlin.com/linux/latest/source/fs/btrfs/file.c)
231 -filesystem, the collector needs to attach `kprobes` and `kretprobes` for each one of the following functions:
311 +To measure the latency of executing some actions in a [btrfs](https://elixir.bootlin.com/linux/latest/source/fs/btrfs/file.c)
312 +filesystem, the collector needs to attach `kprobes` and `kretprobes` for each of the following functions:
313
314 > Note: We are listing two functions used to measure `read` latency, but we use either `btrfs_file_read_iter` or
315 `generic_file_read_iter`, depending on kernel version.
@@ -241,14 +322,14 @@ filesystem, the collector needs to attach `kprobes` and `kretprobes` for each on
322
323 #### File descriptor
324
244 -To give metrics related to `open` and `close` events, instead of attaching kprobes for each syscall used to do these
325 +To give metrics related to `open` and `close` events, instead of attaching kprobes for each syscall used to do these
326 events, the collector attaches `kprobes` for the common function used for syscalls:
327
247 -- [do_sys_open](https://0xax.gitbooks.io/linux-insides/content/SysCall/linux-syscall-5.html ): Internal function used to
328 +- [`do_sys_open`](https://0xax.gitbooks.io/linux-insides/content/SysCall/linux-syscall-5.html ): Internal function used to
329 open files.
249 -- [do_sys_openat2](https://elixir.bootlin.com/linux/v5.6/source/fs/open.c#L1162): Function called from do_sys_open since
250 - version `5.6.0`.
251 -- [close_fd](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg2271761.html): Function used to close file
330 +- [`do_sys_openat2`](https://elixir.bootlin.com/linux/v5.6/source/fs/open.c#L1162):
331 + Function called from `do_sys_open` since version `5.6.0`.
332 +- [`close_fd`](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg2271761.html): Function used to close file
333 descriptor since kernel `5.11.0`.
334 - `__close_fd`: Function used to close files before version `5.11.0`.
335
@@ -256,54 +337,85 @@ events, the collector attaches `kprobes` for the common function used for syscal
337
338 This chart shows the number of times some software tried and failed to open or close a file descriptor.
339
259 -#### VFS Deleted objects
340 +#### VFS
341 +
342 +The Linux Virtual File System (VFS) is an abstraction layer on top of a
343 +concrete filesystem like the ones listed in the parent section, e.g. `ext4`.
344 +
345 +In this section we list the mechanism by which we gather VFS data, and what
346 +charts are consequently created.
347 +
348 +##### VFS eBPF Hooks
349 +
350 +To measure the latency and total quantity of executing some VFS-level
351 +functions, ebpf.plugin needs to attach kprobes and kretprobes for each of the
352 +following functions:
353 +
354 +- `vfs_write`: Function used monitoring the number of successful & failed
355 + filesystem write calls, as well as the total number of written bytes.
356 +- `vfs_writev`: Same function as `vfs_write` but for vector writes (i.e. a
357 + single write operation using a group of buffers rather than 1).
358 +- `vfs_read`: Function used for monitoring the number of successful & failed
359 + filesystem read calls, as well as the total number of read bytes.
360 +- `vfs_readv` Same function as `vfs_read` but for vector reads (i.e. a singe
361 + read operation using a group of buffers rather than 1).
362 +- `vfs_unlink`: Function used for monitoring the number of successful & failed
363 + filesystem unlink calls.
364 +- `vfs_fsync`: Function used for monitoring the number of successful & failed
365 + filesystem fsync calls.
366 +- `vfs_open`: Function used for monitoring the number of successful & failed
367 + filesystem open calls.
368 +- `vfs_create`: Function used for monitoring the number of successful & failed
369 + filesystem create calls.
370 +
371 +##### VFS Deleted objects
372
373 This chart monitors calls to `vfs_unlink`. This function is responsible for removing objects from the file system.
374
263 -#### VFS IO
375 +##### VFS IO
376
377 This chart shows the number of calls to the functions `vfs_read` and `vfs_write`.
378
267 -#### VFS IO bytes
379 +##### VFS IO bytes
380
381 This chart also monitors `vfs_read` and `vfs_write` but, instead of the number of calls, it shows the total amount of
382 bytes read and written with these functions.
383
384 The Agent displays the number of bytes written as negative because they are moving down to disk.
385
274 -#### VFS IO errors
386 +##### VFS IO errors
387
388 The Agent counts and shows the number of instances where a running program experiences a read or write error.
389
278 -#### VFS Create
390 +##### VFS Create
391
392 This chart shows the number of calls to `vfs_create`. This function is responsible for creating files.
393
282 -#### VFS Synchronization
394 +##### VFS Synchronization
395
396 This chart shows the number of calls to `vfs_fsync`. This function is responsible for calling `fsync(2)` or
397 `fdatasync(2)` on a file. You can see more details in the Synchronization section.
398
287 -#### VFS Open
399 +##### VFS Open
400
401 This chart shows the number of calls to `vfs_open`. This function is responsible for opening files.
402
403 #### Directory Cache
404
293 -Metrics for directory cache are collected using kprobe for `lookup_fast`, because we are interested in the number of
294 -times this function is accessed. On the other hand, for `d_lookup` we are not only interested in the number of times it
405 +Metrics for directory cache are collected using kprobe for `lookup_fast`, because we are interested in the number of
406 +times this function is accessed. On the other hand, for `d_lookup` we are not only interested in the number of times it
407 is accessed, but also in possible errors, so we need to attach a `kretprobe`. For this reason, the following is used:
408
297 -- [lookup_fast](https://lwn.net/Articles/649115/): Called to look at data inside the directory cache.
298 -- [d_lookup](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/dcache.c?id=052b398a43a7de8c68c13e7fa05d6b3d16ce6801#n2223):
409 +- [`lookup_fast`](https://lwn.net/Articles/649115/): Called to look at data inside the directory cache.
410 +- [`d_lookup`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/dcache.c?id=052b398a43a7de8c68c13e7fa05d6b3d16ce6801#n2223):
411 Called when the desired file is not inside the directory cache.
412
413 ### Mount Points
414
415 The following `kprobes` are used to collect `mount` & `unmount` call counts:
416
305 -- [mount](https://man7.org/linux/man-pages/man2/mount.2.html): mount filesystem on host.
306 -- [umount](https://man7.org/linux/man-pages/man2/umount.2.html): umount filesystem on host.
417 +- [`mount`](https://man7.org/linux/man-pages/man2/mount.2.html): mount filesystem on host.
418 +- [`umount`](https://man7.org/linux/man-pages/man2/umount.2.html): umount filesystem on host.
419
420 ### Networking Stack
421
@@ -336,9 +448,11 @@ calls, it monitors the number of bytes sent and received.
448
449 ### Apps
450
451 +#### OOM Killing
452 +
453 These are tracepoints related to [OOM](https://en.wikipedia.org/wiki/Out_of_memory) killing processes.
454
341 -- `oom/mark_victm`: Monitors when an oomkill event happens.
455 +- `oom/mark_victim`: Monitors when an oomkill event happens.
456
457 ## Configuration
458
@@ -565,7 +679,7 @@ You can configure the information shown on `outbound` and `inbound` charts with
679 ```
680
681 When you define a `ports` setting, Netdata will collect network metrics for that specific port. For example, if you
568 -write `ports = 19999`, Netdata will collect only connections for itself. The `hostnames` setting accepts
682 +write `ports = 19999`, Netdata will collect only connections for itself. The `hostnames` setting accepts
683 [simple patterns](/libnetdata/simple_pattern/README.md). The `ports`, and `ips` settings accept negation (`!`) to deny
684 specific values or asterisk alone to define all values.
685
@@ -758,7 +872,7 @@ allow unconfined_service_t self:bpf { map_create map_read map_write prog_load pr
872 Then compile your `netdata_ebpf.te` file with the following commands to create a binary that loads the new policies:
873
874 ```bash
761 -# checkmodule -M -m -o netdata_ebpf.mod netdata_ebpf.te
875 +# checkmodule -M -m -o netdata_ebpf.mod netdata_ebpf.te
876 # semodule_package -o netdata_ebpf.pp -m netdata_ebpf.mod
877 ```
878