| 1 | <!--startmeta |
| 2 | custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/ebpf.plugin/integrations/ebpf_oomkill.md" |
| 3 | meta_yaml: "https://github.com/netdata/netdata/edit/master/src/collectors/ebpf.plugin/metadata.yaml" |
| 4 | sidebar_label: "eBPF OOMkill" |
| 5 | learn_status: "Published" |
| 6 | learn_rel_path: "Collecting Metrics/Collectors/Operating Systems" |
| 7 | keywords: ['application', 'memory'] |
| 8 | message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" |
| 9 | endmeta--> |
| 10 | |
| 11 | # eBPF OOMkill |
| 12 | |
| 13 | |
| 14 | <img src="https://netdata.cloud/img/ebpf.jpg" width="150"/> |
| 15 | |
| 16 | |
| 17 | Plugin: ebpf.plugin |
| 18 | Module: oomkill |
| 19 | |
| 20 | <img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> |
| 21 | |
| 22 | ## Overview |
| 23 | |
| 24 | Monitor applications that reach out of memory. |
| 25 | |
| 26 | Attach tracepoint to internal kernel functions. |
| 27 | |
| 28 | This collector is only supported on the following platforms: |
| 29 | |
| 30 | - Linux |
| 31 | |
| 32 | This collector supports collecting metrics from multiple instances of this integration, including remote instances. |
| 33 | |
| 34 | The plugin needs setuid because it loads data inside kernel. Netada sets necessary permission during installation time. |
| 35 | |
| 36 | eBPF OOMkill can be monitored further using the following other integrations: |
| 37 | |
| 38 | - [Applications](/src/collectors/apps.plugin/integrations/applications.md) |
| 39 | - [Containers](/src/collectors/cgroups.plugin/integrations/containers.md) |
| 40 | |
| 41 | ### Default Behavior |
| 42 | |
| 43 | #### Auto-Detection |
| 44 | |
| 45 | The plugin checks kernel compilation flags (CONFIG_KPROBES, CONFIG_BPF, CONFIG_BPF_SYSCALL, CONFIG_BPF_JIT), files inside debugfs, and presence of BTF files to decide which eBPF program will be attached. |
| 46 | |
| 47 | #### Limits |
| 48 | |
| 49 | The default configuration for this integration does not impose any limits on data collection. |
| 50 | |
| 51 | #### Performance Impact |
| 52 | |
| 53 | This thread will add overhead every time that an internal kernel function monitored by this thread is called. |
| 54 | |
| 55 | ## Setup |
| 56 | |
| 57 | |
| 58 | ### Prerequisites |
| 59 | |
| 60 | #### Compile kernel |
| 61 | |
| 62 | Check if your kernel was compiled with necessary options (CONFIG_KPROBES, CONFIG_BPF, CONFIG_BPF_SYSCALL, CONFIG_BPF_JIT) in `/proc/config.gz` or inside /boot/config file. Some cited names can be different accoring preferences of Linux distributions. |
| 63 | When you do not have options set, it is necessary to get the kernel source code from https://kernel.org or a kernel package from your distribution, this last is preferred. The kernel compilation has a well definedd pattern, but distributions can deliver their configuration files |
| 64 | with different names. |
| 65 | |
| 66 | Now follow steps: |
| 67 | 1. Copy the configuration file to /usr/src/linux/.config. |
| 68 | 2. Select the necessary options: make oldconfig |
| 69 | 3. Compile your kernel image: make bzImage |
| 70 | 4. Compile your modules: make modules |
| 71 | 5. Copy your new kernel image for boot loader directory |
| 72 | 6. Install the new modules: make modules_install |
| 73 | 7. Generate an initial ramdisk image (`initrd`) if it is necessary. |
| 74 | 8. Update your boot loader |
| 75 | |
| 76 | |
| 77 | #### Debug Filesystem |
| 78 | |
| 79 | This thread needs to attach a tracepoint to monitor when a process schedule an exit event. To allow this specific feaure, it is necessary to mount `debugfs` (`mount -t debugfs none /sys/kernel/debug/`). |
| 80 | |
| 81 | |
| 82 | |
| 83 | ### Configuration |
| 84 | |
| 85 | #### Options |
| 86 | |
| 87 | Overwrite default configuration reducing number of I/O events |
| 88 | |
| 89 | |
| 90 | |
| 91 | |
| 92 | #### via File |
| 93 | |
| 94 | The configuration file name for this integration is `ebpf.d/oomkill.conf`. |
| 95 | |
| 96 | The file format is a modified INI syntax. The general structure is: |
| 97 | |
| 98 | ```ini |
| 99 | [section1] |
| 100 | option1 = some value |
| 101 | option2 = some other value |
| 102 | |
| 103 | [section2] |
| 104 | option3 = some third value |
| 105 | ``` |
| 106 | You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the |
| 107 | Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory). |
| 108 | |
| 109 | ```bash |
| 110 | cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata |
| 111 | sudo ./edit-config ebpf.d/oomkill.conf |
| 112 | ``` |
| 113 | |
| 114 | ##### Examples |
| 115 | There are no configuration examples. |
| 116 | |
| 117 | |
| 118 | |
| 119 | ## Alerts |
| 120 | |
| 121 | There are no alerts configured by default for this integration. |
| 122 | |
| 123 | |
| 124 | ## Metrics |
| 125 | |
| 126 | Metrics grouped by *scope*. |
| 127 | |
| 128 | The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels. |
| 129 | |
| 130 | |
| 131 | |
| 132 | ### Per cgroup |
| 133 | |
| 134 | These metrics show cgroup/service that reached OOM. |
| 135 | |
| 136 | This scope has no labels. |
| 137 | |
| 138 | Metrics: |
| 139 | |
| 140 | | Metric | Dimensions | Unit | |
| 141 | |:------|:----------|:----| |
| 142 | | cgroup.oomkills | cgroup name | kills | |
| 143 | | services.oomkills | a dimension per systemd service | kills | |
| 144 | |
| 145 | ### Per apps |
| 146 | |
| 147 | These metrics show cgroup/service that reached OOM. |
| 148 | |
| 149 | Labels: |
| 150 | |
| 151 | | Label | Description | |
| 152 | |:-----------|:----------------| |
| 153 | | app_group | The name of the group defined in the configuration. | |
| 154 | |
| 155 | Metrics: |
| 156 | |
| 157 | | Metric | Dimensions | Unit | |
| 158 | |:------|:----------|:----| |
| 159 | | app.oomkill | kills | kills | |
| 160 | |
| 161 | |
| 162 | |
| 163 | ## Troubleshooting |
| 164 | |
| 165 | ### update every |
| 166 | |
| 167 | Data collection frequency. |
| 168 | |
| 169 | ### ebpf load mode |
| 170 | |
| 171 | Define whether plugin will monitor the call (`entry`) for the functions or it will also monitor the return (`return`). |
| 172 | |
| 173 | ### lifetime |
| 174 | |
| 175 | Set default lifetime for thread when enabled by cloud. |
| 176 | |
| 177 |