| 1 | # Monitor Cgroups (cgroups.plugin) |
| 2 | |
| 3 | You can monitor containers and virtual machines using **cgroups**. |
| 4 | |
| 5 | cgroups (or control groups), are a Linux kernel feature that provides accounting and resource usage limiting for |
| 6 | processes. When cgroups are bundled with namespaces (i.e. isolation), they form what we usually call **containers**. |
| 7 | |
| 8 | cgroups are hierarchical, meaning that cgroups can contain child cgroups, which can contain more cgroups, etc. All |
| 9 | accounting is reported (and resource usage limits are applied) also in a hierarchical way. |
| 10 | |
| 11 | To visualize cgroup metrics Netdata provides configuration for cherry-picking the cgroups of interest. By default, |
| 12 | Netdata should pick **systemd services**, all kinds of **containers** (lxc, docker, etc.) and **virtual machines** spawn |
| 13 | by managers that register them with cgroups (qemu, libvirt, etc.). |
| 14 | |
| 15 | ## Supported Technologies |
| 16 | |
| 17 | cgroups.plugin monitors **any process** that creates Linux cgroups. For the following technologies, Netdata also |
| 18 | resolves friendly names (e.g., showing "my-web-app" instead of a raw cgroup path like |
| 19 | `/sys/fs/cgroup/system.slice/docker-abc123.scope`). |
| 20 | |
| 21 | The **Naming** column below indicates whether Netdata can resolve the cgroup path to a human-readable name |
| 22 | (e.g., the container name, VM name, or service name): |
| 23 | |
| 24 | | Technology | Naming | |
| 25 | |:---|:---:| |
| 26 | | Docker | Yes | |
| 27 | | Podman | Yes | |
| 28 | | Kubernetes pods/containers | Yes | |
| 29 | | Nomad (via Docker) | Yes | |
| 30 | | AWS ECS (via Docker) | Yes | |
| 31 | | containerd (via Docker) | Yes | |
| 32 | | Proxmox QEMU/KVM VMs | Yes | |
| 33 | | Proxmox LXC containers | Yes | |
| 34 | | libvirt QEMU/KVM VMs | Yes | |
| 35 | | libvirt LXC containers | Yes | |
| 36 | | LXC 4.0+ (including Incus) | Yes | |
| 37 | | systemd-nspawn | Yes | |
| 38 | | Systemd services | Yes | |
| 39 | | KubeVirt VMs (in K8s) | Partial | |
| 40 | |
| 41 | Technologies that use the above as their underlying infrastructure are also covered. For example: |
| 42 | - **OpenShift** and other Kubernetes distributions (k3s, RKE, RKE2, MicroK8s, EKS, GKE, AKS) — via Kubernetes cgroup paths |
| 43 | - **OpenStack Nova** — via libvirt/QEMU cgroup paths |
| 44 | - **oVirt / RHV** — via libvirt/QEMU cgroup paths |
| 45 | |
| 46 | Any other technology creating Linux cgroups is **monitored** (CPU, memory, disk I/O, network), but will show its |
| 47 | raw cgroup path as the name. |
| 48 | |
| 49 | ## Configuring Netdata for cgroups |
| 50 | |
| 51 | In general, no additional settings are required. Netdata discovers all available cgroups on the host system and |
| 52 | collects their metrics. |
| 53 | |
| 54 | ### How Netdata finds the available cgroups |
| 55 | |
| 56 | Linux exposes resource usage reporting and provides dynamic configuration for cgroups, using virtual files (usually) |
| 57 | under `/sys/fs/cgroup`. Netdata reads `/proc/self/mountinfo` to detect the exact mount point of cgroups. |
| 58 | |
| 59 | Netdata rescans directories inside `/sys/fs/cgroup` for added or removed cgroups every `checking for new cgroups every` |
| 60 | seconds. |
| 61 | |
| 62 | ### Hierarchical search for cgroups |
| 63 | |
| 64 | Since cgroups are hierarchical, for each of the directories shown above, Netdata walks through the subdirectories |
| 65 | recursively searching for cgroups (each subdirectory is another cgroup). |
| 66 | |
| 67 | To provide a sane default for this setting, Netdata uses the following pattern list (patterns starting with `!` give a |
| 68 | negative match and their order is important: the first matching a path will be used): |
| 69 | |
| 70 | ```text |
| 71 | [plugin:cgroups] |
| 72 | search for cgroups in subpaths matching = !*/init.scope !*-qemu !/init.scope !/system !/systemd !/user !/user.slice * |
| 73 | ``` |
| 74 | |
| 75 | So, we disable checking for **child cgroups** in systemd internal |
| 76 | cgroups ([systemd services are monitored by Netdata](#monitoring-systemd-services)), user cgroups (normally used for |
| 77 | desktop and remote user sessions), qemu virtual machines (child cgroups of virtual machines) and `init.scope`. All |
| 78 | others are enabled. |
| 79 | |
| 80 | ### Enabled cgroups |
| 81 | |
| 82 | To provide a sane default, Netdata uses the |
| 83 | following [pattern list](/src/libnetdata/simple_pattern/README.md): |
| 84 | |
| 85 | - Checks the pattern against the path of the cgroup |
| 86 | |
| 87 | ```text |
| 88 | [plugin:cgroups] |
| 89 | enable by default cgroups matching = !*/init.scope *.scope !*/vcpu* !*/emulator !*.mount !*.partition !*.service !*.slice !*.swap !*.user !/ !/docker !/libvirt !/lxc !/lxc/*/ns !/lxc/*/ns/* !/machine !/qemu !/system !/systemd !/user * |
| 90 | ``` |
| 91 | |
| 92 | - Checks the pattern against the name of the cgroup (as you see it on the dashboard) |
| 93 | |
| 94 | ```text |
| 95 | [plugin:cgroups] |
| 96 | enable by default cgroups names matching = * |
| 97 | ``` |
| 98 | |
| 99 | Renaming is configured with the following options: |
| 100 | |
| 101 | ```text |
| 102 | [plugin:cgroups] |
| 103 | run script to rename cgroups matching = *.scope *docker* *lxc* *qemu* !/ !*.mount !*.partition !*.service !*.slice !*.swap !*.user * |
| 104 | script to get cgroup names = /usr/libexec/netdata/plugins.d/cgroup-name.sh |
| 105 | ``` |
| 106 | |
| 107 | The whole point for the additional pattern list, is to limit the number of times the script will be called. Without this |
| 108 | pattern list, the script might be called thousands of times, depending on the number of cgroups available in the system. |
| 109 | |
| 110 | The above pattern list is matched against the path of the cgroup. For matched cgroups, Netdata calls the |
| 111 | script [cgroup-name.sh](https://github.com/netdata/netdata/blob/master/src/collectors/cgroups.plugin/cgroup-name.sh.in) |
| 112 | to get its name. This script queries `docker`, `kubectl`, `podman`, or applies heuristics to find give a name for the |
| 113 | cgroup. |
| 114 | |
| 115 | #### Note on Podman container names |
| 116 | |
| 117 | Podman's security model is a lot more restrictive than Docker's, so Netdata will not be able to detect container names |
| 118 | out of the box unless they were started by the same user as Netdata itself. |
| 119 | |
| 120 | If Podman is used in "rootful" mode, it's also possible to use `podman system service` to grant Netdata access to |
| 121 | container names. To do this, ensure `podman system service` is running and Netdata has access |
| 122 | to `/run/podman/podman.sock` (the default permissions as specified by upstream are `0600`, with owner `root`, so you |
| 123 | will have to adjust the configuration). |
| 124 | |
| 125 | [Docker Socket Proxy (HAProxy)](https://github.com/Tecnativa/docker-socket-proxy) |
| 126 | or [CetusGuard](https://github.com/hectorm/cetusguard) |
| 127 | can also be used to give Netdata restricted access to the socket. Note that `PODMAN_HOST` in Netdata's environment |
| 128 | should be set to the proxy's URL in this case. |
| 129 | |
| 130 | ### Alerts |
| 131 | |
| 132 | CPU and memory limits are watched and used to rise alerts. Memory usage for every cgroup is checked against `ram` |
| 133 | and `ram+swap` limits. CPU usage for every cgroup is checked against `cpuset.cpus` |
| 134 | and `cpu.cfs_period_us` + `cpu.cfs_quota_us` pair assigned for the cgroup. Configuration for the alerts is available |
| 135 | in `health.d/cgroups.conf` file. |
| 136 | |
| 137 | ## Monitoring systemd services |
| 138 | |
| 139 | Netdata monitors **systemd services**. |
| 140 | |
| 141 | Support per distribution: |
| 142 | |
| 143 | | system | charts shown | `/sys/fs/cgroup` tree | comments | |
| 144 | |:----------------:|:------------:|:------------------------------------:|:--------------------------| |
| 145 | | Arch Linux | YES | | | |
| 146 | | Gentoo | NO | | can be enabled, see below | |
| 147 | | Ubuntu 16.04 LTS | YES | | | |
| 148 | | Ubuntu 16.10 | YES | [here](http://pastebin.com/PiWbQEXy) | | |
| 149 | | Fedora 25 | YES | [here](http://pastebin.com/ax0373wF) | | |
| 150 | | Debian 8 | NO | | can be enabled, see below | |
| 151 | | AMI | NO | [here](http://pastebin.com/FrxmptjL) | not a systemd system | |
| 152 | | CentOS 7.3.1611 | NO | [here](http://pastebin.com/SpzgezAg) | can be enabled, see below | |
| 153 | |
| 154 | ### Monitored systemd service metrics |
| 155 | |
| 156 | - CPU utilization |
| 157 | - Used memory |
| 158 | - RSS memory |
| 159 | - Mapped memory |
| 160 | - Cache memory |
| 161 | - Writeback memory |
| 162 | - Memory minor page faults |
| 163 | - Memory major page faults |
| 164 | - Memory charging activity |
| 165 | - Memory uncharging activity |
| 166 | - Memory limit failures |
| 167 | - Swap memory used |
| 168 | - Disk read bandwidth |
| 169 | - Disk write bandwidth |
| 170 | - Disk read operations |
| 171 | - Disk write operations |
| 172 | - Throttle disk read bandwidth |
| 173 | - Throttle disk write bandwidth |
| 174 | - Throttle disk read operations |
| 175 | - Throttle disk write operations |
| 176 | - Queued disk read operations |
| 177 | - Queued disk write operations |
| 178 | - Merged disk read operations |
| 179 | - Merged disk write operations |
| 180 | |
| 181 | ### How to enable cgroup accounting on systemd systems that is by default disabled |
| 182 | |
| 183 | You can verify there is no accounting enabled, by running `systemd-cgtop`. The program will show only resources for |
| 184 | cgroup `/`, but all services will show nothing. |
| 185 | |
| 186 | To enable cgroup accounting, execute this: |
| 187 | |
| 188 | ```sh |
| 189 | sed -e 's|^#Default\(.*\)Accounting=.*$|Default\1Accounting=yes|g' /etc/systemd/system.conf >/tmp/system.conf |
| 190 | ``` |
| 191 | |
| 192 | To see the changes it made, run this: |
| 193 | |
| 194 | ```sh |
| 195 | # diff /etc/systemd/system.conf /tmp/system.conf |
| 196 | 40,44c40,44 |
| 197 | < #DefaultCPUAccounting=no |
| 198 | < #DefaultIOAccounting=no |
| 199 | < #DefaultBlockIOAccounting=no |
| 200 | < #DefaultMemoryAccounting=no |
| 201 | < #DefaultTasksAccounting=yes |
| 202 | --- |
| 203 | > DefaultCPUAccounting=yes |
| 204 | > DefaultIOAccounting=yes |
| 205 | > DefaultBlockIOAccounting=yes |
| 206 | > DefaultMemoryAccounting=yes |
| 207 | > DefaultTasksAccounting=yes |
| 208 | ``` |
| 209 | |
| 210 | If you are happy with the changes, run: |
| 211 | |
| 212 | ```sh |
| 213 | # copy the file to the right location |
| 214 | sudo cp /tmp/system.conf /etc/systemd/system.conf |
| 215 | |
| 216 | # restart systemd to take it into account |
| 217 | sudo systemctl daemon-reexec |
| 218 | ``` |
| 219 | |
| 220 | (`systemctl daemon-reload` does not reload the configuration of the server - so you have to |
| 221 | execute `systemctl daemon-reexec`). |
| 222 | |
| 223 | Now, when you run `systemd-cgtop`, services will start reporting usage (if it does not, restart any service to wake it |
| 224 | up). Refresh your Netdata dashboard, and you will have the charts too. |
| 225 | |
| 226 | In case memory accounting is missing, you will need to enable it at your kernel, by appending the following kernel boot |
| 227 | options and rebooting: |
| 228 | |
| 229 | ```sh |
| 230 | cgroup_enable=memory swapaccount=1 |
| 231 | ``` |
| 232 | |
| 233 | You can add the above, directly at the `linux` line in your `/boot/grub/grub.cfg` or appending them to |
| 234 | the `GRUB_CMDLINE_LINUX` in `/etc/default/grub` (in which case you will have to run `update-grub` before rebooting). On |
| 235 | DigitalOcean debian images you may have to set it at `/etc/default/grub.d/50-cloudimg-settings.cfg`. |
| 236 | |
| 237 | Which systemd services are monitored by Netdata is determined by the following pattern list: |
| 238 | |
| 239 | ```text |
| 240 | [plugin:cgroups] |
| 241 | cgroups to match as systemd services = !/system.slice/*/*.service /system.slice/*.service |
| 242 | ``` |
| 243 | |
| 244 | - - - |
| 245 | |
| 246 | ## Monitoring ephemeral containers |
| 247 | |
| 248 | Netdata monitors containers automatically when it is installed at the host, or when it is installed in a container that |
| 249 | has access to the `/proc` and `/sys` filesystems of the host. |
| 250 | |
| 251 | Network interfaces and cgroups (containers) are self-cleaned. When a network interface or container stops, Netdata might |
| 252 | log a few errors in error.log complaining about files it cannot find, but immediately: |
| 253 | |
| 254 | 1. It will detect this is a removed container or network interface |
| 255 | 2. It will freeze/pause all alerts for them |
| 256 | 3. It will mark their charts as obsolete |
| 257 | 4. Obsolete charts are not be offered on new dashboard sessions (so hit F5 and the charts are gone) |
| 258 | 5. Existing dashboard sessions will continue to see them, but of course they will not refresh |
| 259 | 6. Obsolete charts will be removed from memory, 1 hour after the last user viewed them (configurable) |
| 260 | with `[global].cleanup obsolete charts after seconds = 3600` (at `netdata.conf`). |
| 261 | |
| 262 | ### Monitored container metrics |
| 263 | |
| 264 | - CPU usage |
| 265 | - CPU usage within the limits |
| 266 | - CPU usage per core |
| 267 | - Memory usage |
| 268 | - Writeback memory |
| 269 | - Memory activity |
| 270 | - Memory page faults |
| 271 | - Used memory |
| 272 | - Used RAM within the limits |
| 273 | - Memory utilization |
| 274 | - Memory limit failures |
| 275 | - I/O bandwidth (all disks) |
| 276 | - Serviced I/O operations (all disks) |
| 277 | - Throttle I/O bandwidth (all disks) |
| 278 | - Throttle serviced I/O operations (all disks) |
| 279 | - Queued I/O operations (all disks) |
| 280 | - Merged I/O operations (all disks) |
| 281 | - CPU pressure |
| 282 | - Memory pressure |
| 283 | - Memory full pressure |
| 284 | - I/O pressure |
| 285 | - I/O full pressure |
| 286 | |
| 287 | Network interfaces are monitored by means of |
| 288 | the [proc plugin](/src/collectors/proc.plugin/README.md#monitored-network-interface-metrics). |