feat(cgroups.plugin): add filtering by cgroups names and improve renaming in k8s (#12778)
Ilya Mashchenko committed
May 3, 2022 at 13:24 UTC
ad476d9f66e83f9389abc18025848a49c6266d9e
5 files changed
+779
-543
collectors/cgroups.plugin/README.md
+154
-126
@@ -7,30 +7,28 @@ custom_edit_url: https://github.com/netdata/netdata/edit/master/collectors/cgrou
7
8
You can monitor containers and virtual machines using **cgroups**.
9
10
-cgroups (or control groups), are a Linux kernel feature that provides accounting and resource usage limiting for processes. When cgroups are bundled with namespaces (i.e. isolation), they form what we usually call **containers**.
10
+cgroups (or control groups), are a Linux kernel feature that provides accounting and resource usage limiting for
11
+processes. When cgroups are bundled with namespaces (i.e. isolation), they form what we usually call **containers**.
12
12
-cgroups are hierarchical, meaning that cgroups can contain child cgroups, which can contain more cgroups, etc. All accounting is reported (and resource usage limits are applied) also in a hierarchical way.
13
+cgroups are hierarchical, meaning that cgroups can contain child cgroups, which can contain more cgroups, etc. All
14
+accounting is reported (and resource usage limits are applied) also in a hierarchical way.
15
14
-To visualize cgroup metrics Netdata provides configuration for cherry picking the cgroups of interest. By default (without any configuration) Netdata should pick **systemd services**, all kinds of **containers** (lxc, docker, etc) and **virtual machines** spawn by managers that register them with cgroups (qemu, libvirt, etc).
16
+To visualize cgroup metrics Netdata provides configuration for cherry picking the cgroups of interest. By default (
17
+without any configuration) Netdata should pick **systemd services**, all kinds of **containers** (lxc, docker, etc)
18
+and **virtual machines** spawn by managers that register them with cgroups (qemu, libvirt, etc).
19
16
-## configuring Netdata for cgroups
20
+## Configuring Netdata for cgroups
21
18
-For each cgroup available in the system, Netdata provides this configuration:
19
-
20
-```
21
-[plugin:cgroups]
22
- enable cgroup XXX = yes | no
23
-```
24
-
25
-But it also provides a few patterns to provide a sane default (`yes` or `no`).
26
-
27
-Below we see, how this works.
22
+In general, no additional settings are required. Netdata discovers all available cgroups on the host system and
23
+collects their metrics.
24
25
### how Netdata finds the available cgroups
26
31
-Linux exposes resource usage reporting and provides dynamic configuration for cgroups, using virtual files (usually) under `/sys/fs/cgroup`. Netdata reads `/proc/self/mountinfo` to detect the exact mount point of cgroups. Netdata also allows manual configuration of this mount point, using these settings:
27
+Linux exposes resource usage reporting and provides dynamic configuration for cgroups, using virtual files (usually)
28
+under `/sys/fs/cgroup`. Netdata reads `/proc/self/mountinfo` to detect the exact mount point of cgroups. Netdata also
29
+allows manual configuration of this mount point, using these settings:
30
33
-```
31
+```text
32
[plugin:cgroups]
33
check for new cgroups every = 10
34
path to /sys/fs/cgroup/cpuacct = /sys/fs/cgroup/cpuacct
@@ -43,83 +41,104 @@ Netdata rescans these directories for added or removed cgroups every `check for
41
42
### hierarchical search for cgroups
43
46
-Since cgroups are hierarchical, for each of the directories shown above, Netdata walks through the subdirectories recursively searching for cgroups (each subdirectory is another cgroup).
44
+Since cgroups are hierarchical, for each of the directories shown above, Netdata walks through the subdirectories
45
+recursively searching for cgroups (each subdirectory is another cgroup).
46
48
-To provide a sane default for this setting, Netdata uses the following pattern list (patterns starting with `!` give a negative match and their order is important: the first matching a path will be used):
47
+To provide a sane default for this setting, Netdata uses the following pattern list (patterns starting with `!` give a
48
+negative match and their order is important: the first matching a path will be used):
49
50
-```
50
+```text
51
[plugin:cgroups]
52
search for cgroups in subpaths matching = !*/init.scope !*-qemu !/init.scope !/system !/systemd !/user !/user.slice *
53
```
54
55
-So, we disable checking for **child cgroups** in systemd internal cgroups ([systemd services are monitored by Netdata](#monitoring-systemd-services)), user cgroups (normally used for desktop and remote user sessions), qemu virtual machines (child cgroups of virtual machines) and `init.scope`. All others are enabled.
55
+So, we disable checking for **child cgroups** in systemd internal
56
+cgroups ([systemd services are monitored by Netdata](#monitoring-systemd-services)), user cgroups (normally used for
57
+desktop and remote user sessions), qemu virtual machines (child cgroups of virtual machines) and `init.scope`. All
58
+others are enabled.
59
60
### unified cgroups (cgroups v2) support
61
59
-Basic unified cgroups metrics are supported. To use them instead of v1 cgroups add:
62
+Netdata automatically detects cgroups version. If detection fails Netdata assumes v1.
63
+To switch to v2 manually add:
64
61
-```
65
+```text
66
[plugin:cgroups]
67
use unified cgroups = yes
68
path to unified cgroups = /sys/fs/cgroup
69
```
70
67
-Unified cgroups use same name pattern matching as v1 cgroups. `cgroup_enable_systemd_services_detailed_memory` is currently unsupported when using unified cgroups.
71
+Unified cgroups use same name pattern matching as v1 cgroups. `cgroup_enable_systemd_services_detailed_memory` is
72
+currently unsupported when using unified cgroups.
73
74
### enabled cgroups
75
71
-To check if the cgroup is enabled, Netdata uses this setting:
76
+To provide a sane default, Netdata uses the
77
+following [pattern list](https://learn.netdata.cloud/docs/agent/libnetdata/simple_pattern):
78
73
-```
74
-[plugin:cgroups]
75
- enable cgroup NAME = yes | no
76
-```
79
+- checks the pattern against the path of the cgroup
80
78
-To provide a sane default, Netdata uses the following pattern list (it checks the pattern against the path of the cgroup):
81
+ ```text
82
+ [plugin:cgroups]
83
+ enable by default cgroups matching = !*/init.scope *.scope !*/vcpu* !*/emulator !*.mount !*.partition !*.service !*.slice !*.swap !*.user !/ !/docker !/libvirt !/lxc !/lxc/*/ns !/lxc/*/ns/* !/machine !/qemu !/system !/systemd !/user *
84
+ ```
85
80
-```
81
-[plugin:cgroups]
82
- enable by default cgroups matching = !*/init.scope *.scope !*/vcpu* !*/emulator !*.mount !*.partition !*.service !*.slice !*.swap !*.user !/ !/docker !/libvirt !/lxc !/lxc/*/ns !/lxc/*/ns/* !/machine !/qemu !/system !/systemd !/user *
83
-```
84
-
85
-The above provides the default `yes` or `no` setting for the cgroup. However, there is an additional step. In many cases the cgroups found in the `/sys/fs/cgroup` hierarchy are just random numbers and in many cases these numbers are ephemeral: they change across reboots or sessions.
86
+- checks the pattern against the name of the cgroup (as you see it on the dashboard)
87
87
-So, we need to somehow map the paths of the cgroups to names, to provide consistent Netdata configuration (i.e. there is no point to say `enable cgroup 1234 = yes | no`, if `1234` is a random number that changes over time - we need a name for the cgroup first, so that `enable cgroup NAME = yes | no` will be consistent).
88
+ ```text
89
+ [plugin:cgroups]
90
+ enable by default cgroups names matching = *
91
+ ```
92
89
-For this mapping Netdata provides 2 configuration options:
93
+Renaming is configured with the following options:
94
91
-```
95
+```text
96
[plugin:cgroups]
97
run script to rename cgroups matching = *.scope *docker* *lxc* *qemu* !/ !*.mount !*.partition !*.service !*.slice !*.swap !*.user *
98
script to get cgroup names = /usr/libexec/netdata/plugins.d/cgroup-name.sh
99
```
100
97
-The whole point for the additional pattern list, is to limit the number of times the script will be called. Without this pattern list, the script might be called thousands of times, depending on the number of cgroups available in the system.
101
+The whole point for the additional pattern list, is to limit the number of times the script will be called. Without this
102
+pattern list, the script might be called thousands of times, depending on the number of cgroups available in the system.
103
99
-The above pattern list is matched against the path of the cgroup. For matched cgroups, Netdata calls the script [cgroup-name.sh](https://raw.githubusercontent.com/netdata/netdata/master/collectors/cgroups.plugin/cgroup-name.sh.in) to get its name. This script queries `docker`, `kubectl`, `podman`, or applies heuristics to find give a name for the cgroup.
104
+The above pattern list is matched against the path of the cgroup. For matched cgroups, Netdata calls the
105
+script [cgroup-name.sh](https://raw.githubusercontent.com/netdata/netdata/master/collectors/cgroups.plugin/cgroup-name.sh)
106
+to get its name. This script queries `docker`, `kubectl`, `podman`, or applies heuristics to find give a name for the
107
+cgroup.
108
109
#### Note on Podman container names
110
103
-Podman's security model is a lot more restrictive than Docker's, so Netdata will not be able to detect container names out of the box unless they were started by the same user as Netdata itself.
111
+Podman's security model is a lot more restrictive than Docker's, so Netdata will not be able to detect container names
112
+out of the box unless they were started by the same user as Netdata itself.
113
105
-If Podman is used in "rootful" mode, it's also possible to use `podman system service` to grant Netdata access to container names. To do this, ensure `podman system service` is running and Netdata has access to `/run/podman/podman.sock` (the default permissions as specified by upstream are `0600`, with owner `root`, so you will have to adjust the configuration).
114
+If Podman is used in "rootful" mode, it's also possible to use `podman system service` to grant Netdata access to
115
+container names. To do this, ensure `podman system service` is running and Netdata has access
116
+to `/run/podman/podman.sock` (the default permissions as specified by upstream are `0600`, with owner `root`, so you
117
+will have to adjust the configuration).
118
107
-[docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy) can also be used to give Netdata restricted access to the socket. Note that `PODMAN_HOST` in Netdata's environment should be set to the proxy's URL in this case.
119
+[docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy) can also be used to give Netdata restricted
120
+access to the socket. Note that `PODMAN_HOST` in Netdata's environment should be set to the proxy's URL in this case.
121
122
### charts with zero metrics
123
111
-By default, Netdata will enable monitoring metrics only when they are not zero. If they are constantly zero they are ignored. Metrics that will start having values, after Netdata is started, will be detected and charts will be automatically added to the dashboard (a refresh of the dashboard is needed for them to appear though). Set `yes` for a chart instead of `auto` to enable it permanently. For example:
124
+By default, Netdata will enable monitoring metrics only when they are not zero. If they are constantly zero they are
125
+ignored. Metrics that will start having values, after Netdata is started, will be detected and charts will be
126
+automatically added to the dashboard (a refresh of the dashboard is needed for them to appear though). Set `yes` for a
127
+chart instead of `auto` to enable it permanently. For example:
128
113
-```
129
+```text
130
[plugin:cgroups]
131
enable memory (used mem including cache) = yes
132
```
133
118
-You can also set the `enable zero metrics` option to `yes` in the `[global]` section which enables charts with zero metrics for all internal Netdata plugins.
134
+You can also set the `enable zero metrics` option to `yes` in the `[global]` section which enables charts with zero
135
+metrics for all internal Netdata plugins.
136
137
### alarms
138
122
-CPU and memory limits are watched and used to rise alarms. Memory usage for every cgroup is checked against `ram` and `ram+swap` limits. CPU usage for every cgroup is checked against `cpuset.cpus` and `cpu.cfs_period_us` + `cpu.cfs_quota_us` pair assigned for the cgroup. Configuration for the alarms is available in `health.d/cgroups.conf` file.
139
+CPU and memory limits are watched and used to rise alarms. Memory usage for every cgroup is checked against `ram`
140
+and `ram+swap` limits. CPU usage for every cgroup is checked against `cpuset.cpus` and `cpu.cfs_period_us` + `cpu.cfs_quota_us` pair assigned for the cgroup. Configuration for the alarms is available in `health.d/cgroups.conf`
141
+file.
142
143
## Monitoring systemd services
144
@@ -129,47 +148,48 @@ Netdata monitors **systemd services**. Example:
148
149
Support per distribution:
150
132
-|system|systemd services<br/>charts shown|`tree`<br/>`/sys/fs/cgroup`|comments|
133
-|:----:|:-------------------------------:|:-------------------------:|:-------|
134
-|Arch Linux|YES|||
135
-|Gentoo|NO||can be enabled, see below|
136
-|Ubuntu 16.04 LTS|YES|||
137
-|Ubuntu 16.10|YES|[here](http://pastebin.com/PiWbQEXy)||
138
-|Fedora 25|YES|[here](http://pastebin.com/ax0373wF)||
139
-|Debian 8|NO||can be enabled, see below|
140
-|AMI|NO|[here](http://pastebin.com/FrxmptjL)|not a systemd system|
141
-|CentOS 7.3.1611|NO|[here](http://pastebin.com/SpzgezAg)|can be enabled, see below|
151
+| system | charts shown | `/sys/fs/cgroup` tree | comments |
152
+|:----------------:|:------------:|:------------------------------------:|:--------------------------|
153
+| Arch Linux | YES | | |
154
+| Gentoo | NO | | can be enabled, see below |
155
+| Ubuntu 16.04 LTS | YES | | |
156
+| Ubuntu 16.10 | YES | [here](http://pastebin.com/PiWbQEXy) | |
157
+| Fedora 25 | YES | [here](http://pastebin.com/ax0373wF) | |
158
+| Debian 8 | NO | | can be enabled, see below |
159
+| AMI | NO | [here](http://pastebin.com/FrxmptjL) | not a systemd system |
160
+| CentOS 7.3.1611 | NO | [here](http://pastebin.com/SpzgezAg) | can be enabled, see below |
161
162
### Monitored systemd service metrics
163
145
-- CPU utilization
146
-- Used memory
147
-- RSS memory
148
-- Mapped memory
149
-- Cache memory
150
-- Writeback memory
151
-- Memory minor page faults
152
-- Memory major page faults
153
-- Memory charging activity
154
-- Memory uncharging activity
155
-- Memory limit failures
156
-- Swap memory used
157
-- Disk read bandwidth
158
-- Disk write bandwidth
159
-- Disk read operations
160
-- Disk write operations
161
-- Throttle disk read bandwidth
162
-- Throttle disk write bandwidth
163
-- Throttle disk read operations
164
-- Throttle disk write operations
165
-- Queued disk read operations
166
-- Queued disk write operations
167
-- Merged disk read operations
168
-- Merged disk write operations
164
+- CPU utilization
165
+- Used memory
166
+- RSS memory
167
+- Mapped memory
168
+- Cache memory
169
+- Writeback memory
170
+- Memory minor page faults
171
+- Memory major page faults
172
+- Memory charging activity
173
+- Memory uncharging activity
174
+- Memory limit failures
175
+- Swap memory used
176
+- Disk read bandwidth
177
+- Disk write bandwidth
178
+- Disk read operations
179
+- Disk write operations
180
+- Throttle disk read bandwidth
181
+- Throttle disk write bandwidth
182
+- Throttle disk read operations
183
+- Throttle disk write operations
184
+- Queued disk read operations
185
+- Queued disk write operations
186
+- Merged disk read operations
187
+- Merged disk write operations
188
189
### how to enable cgroup accounting on systemd systems that is by default disabled
190
172
-You can verify there is no accounting enabled, by running `systemd-cgtop`. The program will show only resources for cgroup `/`, but all services will show nothing.
191
+You can verify there is no accounting enabled, by running `systemd-cgtop`. The program will show only resources for
192
+cgroup `/`, but all services will show nothing.
193
194
To enable cgroup accounting, execute this:
195
@@ -179,7 +199,7 @@ sed -e 's|^#Default\(.*\)Accounting=.*$|Default\1Accounting=yes|g' /etc/systemd/
199
200
To see the changes it made, run this:
201
182
-```
202
+```sh
203
# diff /etc/systemd/system.conf /tmp/system.conf
204
40,44c40,44
205
< #DefaultCPUAccounting=no
@@ -205,21 +225,25 @@ sudo cp /tmp/system.conf /etc/systemd/system.conf
225
sudo systemctl daemon-reexec
226
```
227
208
-(`systemctl daemon-reload` does not reload the configuration of the server - so you have to execute `systemctl daemon-reexec`).
228
+(`systemctl daemon-reload` does not reload the configuration of the server - so you have to
229
+execute `systemctl daemon-reexec`).
230
210
-Now, when you run `systemd-cgtop`, services will start reporting usage (if it does not, restart a service - any service - to wake it up). Refresh your Netdata dashboard, and you will have the charts too.
231
+Now, when you run `systemd-cgtop`, services will start reporting usage (if it does not, restart any service to wake it up). Refresh your Netdata dashboard, and you will have the charts too.
232
212
-In case memory accounting is missing, you will need to enable it at your kernel, by appending the following kernel boot options and rebooting:
233
+In case memory accounting is missing, you will need to enable it at your kernel, by appending the following kernel boot
234
+options and rebooting:
235
214
-```
236
+```sh
237
cgroup_enable=memory swapaccount=1
238
```
239
218
-You can add the above, directly at the `linux` line in your `/boot/grub/grub.cfg` or appending them to the `GRUB_CMDLINE_LINUX` in `/etc/default/grub` (in which case you will have to run `update-grub` before rebooting). On DigitalOcean debian images you may have to set it at `/etc/default/grub.d/50-cloudimg-settings.cfg`.
240
+You can add the above, directly at the `linux` line in your `/boot/grub/grub.cfg` or appending them to
241
+the `GRUB_CMDLINE_LINUX` in `/etc/default/grub` (in which case you will have to run `update-grub` before rebooting). On
242
+DigitalOcean debian images you may have to set it at `/etc/default/grub.d/50-cloudimg-settings.cfg`.
243
244
Which systemd services are monitored by Netdata is determined by the following pattern list:
245
222
-```
246
+```text
247
[plugin:cgroups]
248
cgroups to match as systemd services = !/system.slice/*/*.service /system.slice/*.service
249
```
@@ -228,53 +252,57 @@ Which systemd services are monitored by Netdata is determined by the following p
252
253
## Monitoring ephemeral containers
254
231
-Netdata monitors containers automatically when it is installed at the host, or when it is installed in a container that has access to the `/proc` and `/sys` filesystems of the host.
255
+Netdata monitors containers automatically when it is installed at the host, or when it is installed in a container that
256
+has access to the `/proc` and `/sys` filesystems of the host.
257
258
Netdata prior to v1.6 had 2 issues when such containers were monitored:
259
235
-1. network interface alarms where triggering when containers were stopped
260
+1. network interface alarms where triggering when containers were stopped
261
237
-2. charts were never cleaned up, so after some time dozens of containers were showing up on the dashboard, and they were occupying memory.
262
+2. charts were never cleaned up, so after some time dozens of containers were showing up on the dashboard, and they were
263
+ occupying memory.
264
265
### the current Netdata
266
267
network interfaces and cgroups (containers) are now self-cleaned.
268
243
-So, when a network interface or container stops, Netdata might log a few errors in error.log complaining about files it cannot find, but immediately:
269
+So, when a network interface or container stops, Netdata might log a few errors in error.log complaining about files it
270
+cannot find, but immediately:
271
245
-1. it will detect this is a removed container or network interface
246
-2. it will freeze/pause all alarms for them
247
-3. it will mark their charts as obsolete
248
-4. obsolete charts are not be offered on new dashboard sessions (so hit F5 and the charts are gone)
249
-5. existing dashboard sessions will continue to see them, but of course they will not refresh
250
-6. obsolete charts will be removed from memory, 1 hour after the last user viewed them (configurable with `[global].cleanup obsolete charts after seconds = 3600` (at `netdata.conf`).
251
-7. when obsolete charts are removed from memory they are also deleted from disk (configurable with `[global].delete obsolete charts files = yes`)
272
+1. it will detect this is a removed container or network interface
273
+2. it will freeze/pause all alarms for them
274
+3. it will mark their charts as obsolete
275
+4. obsolete charts are not be offered on new dashboard sessions (so hit F5 and the charts are gone)
276
+5. existing dashboard sessions will continue to see them, but of course they will not refresh
277
+6. obsolete charts will be removed from memory, 1 hour after the last user viewed them (configurable
278
+ with `[global].cleanup obsolete charts after seconds = 3600` (at `netdata.conf`).
279
+7. when obsolete charts are removed from memory they are also deleted from disk (configurable
280
+ with `[global].delete obsolete charts files = yes`)
281
282
### Monitored container metrics
283
255
-- CPU usage
256
-- CPU usage within the limits
257
-- CPU usage per core
258
-- Memory usage
259
-- Writeback memory
260
-- Memory activity
261
-- Memory page faults
262
-- Used memory
263
-- Used RAM within the limits
264
-- Memory utilization
265
-- Memory limit failures
266
-- I/O bandwidth (all disks)
267
-- Serviced I/O operations (all disks)
268
-- Throttle I/O bandwidth (all disks)
269
-- Throttle serviced I/O operations (all disks)
270
-- Queued I/O operations (all disks)
271
-- Merged I/O operations (all disks)
272
-- CPU pressure
273
-- Memory pressure
274
-- Memory full pressure
275
-- I/O pressure
276
-- I/O full pressure
277
-
278
-Network interfaces are monitored by means of the [proc plugin](/collectors/proc.plugin/README.md#monitored-network-interface-metrics).
279
-
280
-
284
+- CPU usage
285
+- CPU usage within the limits
286
+- CPU usage per core
287
+- Memory usage
288
+- Writeback memory
289
+- Memory activity
290
+- Memory page faults
291
+- Used memory
292
+- Used RAM within the limits
293
+- Memory utilization
294
+- Memory limit failures
295
+- I/O bandwidth (all disks)
296
+- Serviced I/O operations (all disks)
297
+- Throttle I/O bandwidth (all disks)
298
+- Throttle serviced I/O operations (all disks)
299
+- Queued I/O operations (all disks)
300
+- Merged I/O operations (all disks)
301
+- CPU pressure
302
+- Memory pressure
303
+- Memory full pressure
304
+- I/O pressure
305
+- I/O full pressure
306
+
307
+Network interfaces are monitored by means of
308
+the [proc plugin](/collectors/proc.plugin/README.md#monitored-network-interface-metrics).
collectors/cgroups.plugin/cgroup-name.sh
+133
-77
@@ -114,6 +114,31 @@ function add_lbl_prefix() {
114
echo "${new_labels:0:-1}" # trim last ','
115
}
116
117
+function k8s_is_pause_container() {
118
+ local cgroup_path="${1}"
119
+
120
+ local file
121
+ if [ -d "${NETDATA_HOST_PREFIX}/sys/fs/cgroup/cpuacct" ]; then
122
+ file="${NETDATA_HOST_PREFIX}/sys/fs/cgroup/cpuacct/$cgroup_path/cgroup.procs"
123
+ else
124
+ file="${NETDATA_HOST_PREFIX}/sys/fs/cgroup/$cgroup_path/cgroup.procs"
125
+ fi
126
+
127
+ [ ! -f "$file" ] && return 1
128
+
129
+ local procs
130
+ IFS= read -rd' ' procs 2>/dev/null <"$file"
131
+ #shellcheck disable=SC2206
132
+ procs=($procs)
133
+
134
+ [ "${#procs[@]}" -ne 1 ] && return 1
135
+
136
+ IFS= read -r comm 2>/dev/null <"/proc/${procs[0]}/comm"
137
+
138
+ [ "$comm" == "pause" ]
139
+ return
140
+}
141
+
142
# k8s_get_kubepod_name resolves */kubepods/* cgroup name.
143
# pod level cgroup name format: 'pod_<namespace>_<pod_name>'
144
# container level cgroup name format: 'cntr_<namespace>_<pod_name>_<container_name>'
@@ -151,7 +176,8 @@ function k8s_get_kubepod_name() {
176
# - replaces '.' with '-'
177
178
local fn="${FUNCNAME[0]}"
154
- local id="${1}"
179
+ local cgroup_path="${1}"
180
+ local id="${2}"
181
182
if [[ ! $id =~ ^kubepods ]]; then
183
warning "${fn}: '${id}' is not kubepod cgroup."
@@ -195,82 +221,95 @@ function k8s_get_kubepod_name() {
221
[ -n "$pod_uid" ] && info "${fn}: cgroup '$id' is a pod(uid:$pod_uid)"
222
[ -n "$cntr_id" ] && info "${fn}: cgroup '$id' is a container(id:$cntr_id)"
223
224
+ if [ -n "$cntr_id" ] && k8s_is_pause_container "$cgroup_path"; then
225
+ return 1
226
+ fi
227
+
228
if ! command -v jq > /dev/null 2>&1; then
229
warning "${fn}: 'jq' command not available."
230
return 1
231
fi
232
203
- local kube_system_ns
204
- local tmp_kube_system_ns_file="${TMPDIR:-"/tmp/"}netdata-cgroups-kube-system-ns"
205
- [ -f "$tmp_kube_system_ns_file" ] && kube_system_ns=$(cat "$tmp_kube_system_ns_file" 2> /dev/null)
233
+ local tmp_kube_system_ns_uid_file="${TMPDIR:-"/tmp"}/netdata-cgroups-kubesystem-uid"
234
+ local tmp_kube_containers_file="${TMPDIR:-"/tmp"}/netdata-cgroups-containers"
235
+
236
+ local kube_system_uid
237
+ local labels
238
207
- local pods
208
- if [ -n "${KUBERNETES_SERVICE_HOST}" ] && [ -n "${KUBERNETES_PORT_443_TCP_PORT}" ]; then
209
- local token header host url
210
- token="$(< /var/run/secrets/kubernetes.io/serviceaccount/token)"
211
- header="Authorization: Bearer $token"
212
- host="$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT"
239
+ if [ -n "$cntr_id" ] &&
240
+ [ -f "$tmp_kube_system_ns_uid_file" ] &&
241
+ [ -f "$tmp_kube_containers_file" ] &&
242
+ labels=$(grep "$cntr_id" "$tmp_kube_containers_file" 2>/dev/null); then
243
+ IFS= read -r kube_system_uid 2>/dev/null <"$tmp_kube_system_ns_uid_file"
244
+ else
245
+ IFS= read -r kube_system_uid 2>/dev/null <"$tmp_kube_system_ns_uid_file"
246
+ local kube_system_ns
247
+ local pods
248
+ if [ -n "${KUBERNETES_SERVICE_HOST}" ] && [ -n "${KUBERNETES_PORT_443_TCP_PORT}" ]; then
249
+ local token header host url
250
+ token="$(</var/run/secrets/kubernetes.io/serviceaccount/token)"
251
+ header="Authorization: Bearer $token"
252
+ host="$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT"
253
+
254
+ if [ -z "$kube_system_uid" ]; then
255
+ url="https://$host/api/v1/namespaces/kube-system"
256
+ # FIX: check HTTP response code
257
+ if ! kube_system_ns=$(curl -sSk -H "$header" "$url" 2>&1); then
258
+ warning "${fn}: error on curl '${url}': ${kube_system_ns}."
259
+ fi
260
+ fi
261
214
- if [ -z "$kube_system_ns" ]; then
215
- url="https://$host/api/v1/namespaces/kube-system"
262
+ url="https://$host/api/v1/pods"
263
+ [ -n "$MY_NODE_NAME" ] && url+="?fieldSelector=spec.nodeName==$MY_NODE_NAME"
264
# FIX: check HTTP response code
217
- if ! kube_system_ns=$(curl -sSk -H "$header" "$url" 2>&1); then
218
- warning "${fn}: error on curl '${url}': ${kube_system_ns}."
219
- else
220
- echo "$kube_system_ns" > "$tmp_kube_system_ns_file" 2> /dev/null
265
+ if ! pods=$(curl -sSk -H "$header" "$url" 2>&1); then
266
+ warning "${fn}: error on curl '${url}': ${pods}."
267
+ return 1
268
+ fi
269
+ elif ps -C kubelet >/dev/null 2>&1 && command -v kubectl >/dev/null 2>&1; then
270
+ if [ -z "$kube_system_uid" ]; then
271
+ if ! kube_system_ns=$(kubectl get namespaces kube-system -o json 2>&1); then
272
+ warning "${fn}: error on 'kubectl': ${kube_system_ns}."
273
+ fi
274
fi
222
- fi
275
224
- url="https://$host/api/v1/pods"
225
- [ -n "$MY_NODE_NAME" ] && url+="?fieldSelector=spec.nodeName==$MY_NODE_NAME"
226
- # FIX: check HTTP response code
227
- if ! pods=$(curl -sSk -H "$header" "$url" 2>&1); then
228
- warning "${fn}: error on curl '${url}': ${pods}."
276
+ [[ -z ${KUBE_CONFIG+x} ]] && KUBE_CONFIG="/etc/kubernetes/admin.conf"
277
+ if ! pods=$(kubectl --kubeconfig="$KUBE_CONFIG" get pods --all-namespaces -o json 2>&1); then
278
+ warning "${fn}: error on 'kubectl': ${pods}."
279
+ return 1
280
+ fi
281
+ else
282
+ warning "${fn}: not inside the k8s cluster and 'kubectl' command not available."
283
return 1
284
fi
231
- elif ps -C kubelet > /dev/null 2>&1 && command -v kubectl > /dev/null 2>&1; then
232
- if [ -z "$kube_system_ns" ]; then
233
- if ! kube_system_ns=$(kubectl get namespaces kube-system -o json 2>&1); then
234
- warning "${fn}: error on 'kubectl': ${kube_system_ns}."
235
- else
236
- echo "$kube_system_ns" > "$tmp_kube_system_ns_file" 2> /dev/null
237
- fi
285
+
286
+ if [ -n "$kube_system_ns" ] && ! kube_system_uid=$(jq -r '.metadata.uid' <<<"$kube_system_ns" 2>&1); then
287
+ warning "${fn}: error on 'jq' parse kube_system_ns: ${kube_system_uid}."
288
fi
289
240
- [[ -z ${KUBE_CONFIG+x} ]] && KUBE_CONFIG="/etc/kubernetes/admin.conf"
241
- if ! pods=$(kubectl --kubeconfig="$KUBE_CONFIG" get pods --all-namespaces -o json 2>&1); then
242
- warning "${fn}: error on 'kubectl': ${pods}."
290
+ local jq_filter
291
+ jq_filter+='.items[] | "'
292
+ jq_filter+='namespace=\"\(.metadata.namespace)\",'
293
+ jq_filter+='pod_name=\"\(.metadata.name)\",'
294
+ jq_filter+='pod_uid=\"\(.metadata.uid)\",'
295
+ #jq_filter+='\(.metadata.labels | to_entries | map("pod_label_"+.key+"=\""+.value+"\"") | join(",") | if length > 0 then .+"," else . end)'
296
+ jq_filter+='\((.metadata.ownerReferences[]? | select(.controller==true) | "controller_kind=\""+.kind+"\",controller_name=\""+.name+"\",") // "")'
297
+ jq_filter+='node_name=\"\(.spec.nodeName)\",'
298
+ jq_filter+='" + '
299
+ jq_filter+='(.status.containerStatuses[]? | "'
300
+ jq_filter+='container_name=\"\(.name)\",'
301
+ jq_filter+='container_id=\"\(.containerID)\"'
302
+ jq_filter+='") | '
303
+ jq_filter+='sub("(docker|cri-o|containerd)://";"")' # containerID: docker://a346da9bc0e3eaba6b295f64ac16e02f2190db2cef570835706a9e7a36e2c722
304
+
305
+ local containers
306
+ if ! containers=$(jq -r "${jq_filter}" <<<"$pods" 2>&1); then
307
+ warning "${fn}: error on 'jq' parse pods: ${containers}."
308
return 1
309
fi
245
- else
246
- warning "${fn}: not inside the k8s cluster and 'kubectl' command not available."
247
- return 1
248
- fi
310
250
- local kube_system_uid
251
- if [ -n "$kube_system_ns" ] && ! kube_system_uid=$(jq -r '.metadata.uid' <<< "$kube_system_ns" 2>&1); then
252
- warning "${fn}: error on 'jq' parse kube_system_ns: ${kube_system_uid}."
253
- fi
254
-
255
- local jq_filter
256
- jq_filter+='.items[] | "'
257
- jq_filter+='namespace=\"\(.metadata.namespace)\",'
258
- jq_filter+='pod_name=\"\(.metadata.name)\",'
259
- jq_filter+='pod_uid=\"\(.metadata.uid)\",'
260
- #jq_filter+='\(.metadata.labels | to_entries | map("pod_label_"+.key+"=\""+.value+"\"") | join(",") | if length > 0 then .+"," else . end)'
261
- jq_filter+='\((.metadata.ownerReferences[]? | select(.controller==true) | "controller_kind=\""+.kind+"\",controller_name=\""+.name+"\",") // "")'
262
- jq_filter+='node_name=\"\(.spec.nodeName)\",'
263
- jq_filter+='" + '
264
- jq_filter+='(.status.containerStatuses[]? | "'
265
- jq_filter+='container_name=\"\(.name)\",'
266
- jq_filter+='container_id=\"\(.containerID)\"'
267
- jq_filter+='") | '
268
- jq_filter+='sub("(docker|cri-o|containerd)://";"")' # containerID: docker://a346da9bc0e3eaba6b295f64ac16e02f2190db2cef570835706a9e7a36e2c722
269
-
270
- local containers
271
- if ! containers=$(jq -r "${jq_filter}" <<< "$pods" 2>&1); then
272
- warning "${fn}: error on 'jq' parse pods: ${containers}."
273
- return 1
311
+ [ -n "$kube_system_ns" ] && [ -n "$kube_system_uid" ] && echo "$kube_system_uid" >"$tmp_kube_system_ns_uid_file" 2>/dev/null
312
+ echo "$containers" >"$tmp_kube_containers_file" 2>/dev/null
313
fi
314
315
local qos_class
@@ -282,9 +321,8 @@ function k8s_get_kubepod_name() {
321
322
# available labels:
323
# namespace, pod_name, pod_uid, container_name, container_id, node_name
285
- local labels
324
if [ -n "$cntr_id" ]; then
287
- if labels=$(grep "$cntr_id" <<< "$containers" 2> /dev/null); then
325
+ if [ -n "$labels" ] || labels=$(grep "$cntr_id" <<< "$containers" 2> /dev/null); then
326
labels+=',kind="container"'
327
labels+=",qos_class=\"$qos_class\""
328
[ -n "$kube_system_uid" ] && [ "$kube_system_uid" != "null" ] && labels+=",cluster_id=\"$kube_system_uid\""
@@ -294,6 +332,8 @@ function k8s_get_kubepod_name() {
332
name+="_$(get_lbl_val "$labels" container_name)"
333
labels=$(add_lbl_prefix "$labels" "k8s_")
334
name+=" $labels"
335
+ else
336
+ return 2
337
fi
338
elif [ -n "$pod_uid" ]; then
339
if labels=$(grep "$pod_uid" -m 1 <<< "$containers" 2> /dev/null); then
@@ -306,6 +346,8 @@ function k8s_get_kubepod_name() {
346
name+="_$(get_lbl_val "$labels" pod_name)"
347
labels=$(add_lbl_prefix "$labels" "k8s_")
348
name+=" $labels"
349
+ else
350
+ return 2
351
fi
352
fi
353
@@ -322,15 +364,13 @@ function k8s_get_kubepod_name() {
364
365
function k8s_get_name() {
366
local fn="${FUNCNAME[0]}"
325
- local id="${1}"
326
-
327
- NAME=$(k8s_get_kubepod_name "$id")
367
+ local cgroup_path="${1}"
368
+ local id="${2}"
369
329
- if [ -z "${NAME}" ]; then
330
- warning "${fn}: cannot find the name of cgroup with id '${id}'. Setting name to ${id} and disabling it."
331
- NAME="${id}"
332
- NAME_NOT_FOUND=3
333
- else
370
+ NAME=$(k8s_get_kubepod_name "$cgroup_path" "$id")
371
+
372
+ case "$?" in
373
+ 0)
374
NAME="k8s_${NAME}"
375
376
local name labels
@@ -341,7 +381,19 @@ function k8s_get_name() {
381
else
382
info "${fn}: cgroup '${id}' has chart name '${NAME}'"
383
fi
344
- fi
384
+ EXIT_CODE=$EXIT_SUCCESS
385
+ ;;
386
+ 2)
387
+ warning "${fn}: cannot find the name of cgroup with id '${id}'. Setting name to ${id} and asking for retry."
388
+ NAME="${id}"
389
+ EXIT_CODE=$EXIT_RETRY
390
+ ;;
391
+ *)
392
+ warning "${fn}: cannot find the name of cgroup with id '${id}'. Setting name to ${id} and disabling it."
393
+ NAME="${id}"
394
+ EXIT_CODE=$EXIT_DISABLE
395
+ ;;
396
+ esac
397
}
398
399
function docker_get_name() {
@@ -353,7 +405,7 @@ function docker_get_name() {
405
fi
406
if [ -z "${NAME}" ]; then
407
warning "cannot find the name of docker container '${id}'"
356
- NAME_NOT_FOUND=2
408
+ EXIT_CODE=$EXIT_RETRY
409
NAME="${id:0:12}"
410
else
411
info "docker container '${id}' is named '${NAME}'"
@@ -378,7 +430,7 @@ function podman_get_name() {
430
431
if [ -z "${NAME}" ]; then
432
warning "cannot find the name of podman container '${id}'"
381
- NAME_NOT_FOUND=2
433
+ EXIT_CODE=$EXIT_RETRY
434
NAME="${id:0:12}"
435
else
436
info "podman container '${id}' is named '${NAME}'"
@@ -398,8 +450,12 @@ function podman_validate_id() {
450
451
DOCKER_HOST="${DOCKER_HOST:=/var/run/docker.sock}"
452
PODMAN_HOST="${PODMAN_HOST:=/run/podman/podman.sock}"
401
-CGROUP="${1}"
402
-NAME_NOT_FOUND=0
453
+CGROUP_PATH="${1}" # the path as it is (e.g. '/docker/efcf4c409')
454
+CGROUP="${2}" # the modified path (e.g. 'docker_efcf4c409')
455
+EXIT_SUCCESS=0
456
+EXIT_RETRY=2
457
+EXIT_DISABLE=3
458
+EXIT_CODE=$EXIT_SUCCESS
459
NAME=
460
461
# -----------------------------------------------------------------------------
@@ -410,7 +466,7 @@ fi
466
467
if [ -z "${NAME}" ]; then
468
if [[ ${CGROUP} =~ ^.*kubepods.* ]]; then
413
- k8s_get_name "${CGROUP}"
469
+ k8s_get_name "${CGROUP_PATH}" "${CGROUP}"
470
fi
471
fi
472
@@ -481,4 +537,4 @@ fi
537
info "cgroup '${CGROUP}' is called '${NAME}'"
538
echo "${NAME}"
539
484
-exit ${NAME_NOT_FOUND}
540
+exit ${EXIT_CODE}
collectors/cgroups.plugin/sys_fs_cgroup.c
+487
-335
@@ -9,6 +9,8 @@
9
// ----------------------------------------------------------------------------
10
// cgroup globals
11
12
+static int is_inside_k8s = 0;
13
+
14
static long system_page_size = 4096; // system will be queried via sysconf() in configuration()
15
16
static int cgroup_enable_cpuacct_stat = CONFIG_BOOLEAN_AUTO;
@@ -41,6 +43,7 @@ static int cgroup_unified_exist = CONFIG_BOOLEAN_AUTO;
43
static int cgroup_search_in_devices = 1;
44
45
static int cgroup_enable_new_cgroups_detected_at_runtime = 1;
46
+
47
static int cgroup_check_for_new_every = 10;
48
static int cgroup_update_every = 1;
49
static int cgroup_containers_chart_priority = NETDATA_CHART_PRIO_CGROUPS_CONTAINERS;
@@ -60,8 +63,9 @@ static int cgroup_root_count = 0;
63
static int cgroup_root_max = 1000;
64
static int cgroup_max_depth = 0;
65
63
-static SIMPLE_PATTERN *enabled_cgroup_patterns = NULL;
66
static SIMPLE_PATTERN *enabled_cgroup_paths = NULL;
67
+static SIMPLE_PATTERN *enabled_cgroup_names = NULL;
68
+static SIMPLE_PATTERN *search_cgroup_paths = NULL;
69
static SIMPLE_PATTERN *enabled_cgroup_renames = NULL;
70
static SIMPLE_PATTERN *systemd_services_cgroups = NULL;
71
@@ -411,7 +415,7 @@ void read_cgroup_plugin_configuration() {
415
416
cgroup_enable_new_cgroups_detected_at_runtime = config_get_boolean("plugin:cgroups", "enable new cgroups detected at run time", cgroup_enable_new_cgroups_detected_at_runtime);
417
414
- enabled_cgroup_patterns = simple_pattern_create(
418
+ enabled_cgroup_paths = simple_pattern_create(
419
config_get("plugin:cgroups", "enable by default cgroups matching",
420
// ----------------------------------------------------------------
421
@@ -453,7 +457,12 @@ void read_cgroup_plugin_configuration() {
457
" * " // enable anything else
458
), NULL, SIMPLE_PATTERN_EXACT);
459
456
- enabled_cgroup_paths = simple_pattern_create(
460
+ enabled_cgroup_names = simple_pattern_create(
461
+ config_get("plugin:cgroups", "enable by default cgroups names matching",
462
+ " * "
463
+ ), NULL, SIMPLE_PATTERN_EXACT);
464
+
465
+ search_cgroup_paths = simple_pattern_create(
466
config_get("plugin:cgroups", "search for cgroups in subpaths matching",
467
" !*/init.scope " // ignore init.scope
468
" !*-qemu " // #345
@@ -730,6 +739,9 @@ struct cgroup_network_interface {
739
struct cgroup {
740
uint32_t options;
741
742
+ int first_time_seen; // first time seen by the discoverer
743
+ int processed; // the discoverer is done processing a cgroup (resolved name, set 'enabled' option)
744
+
745
char available; // found in the filesystem
746
char enabled; // enabled in the config
747
@@ -859,7 +871,114 @@ struct discovery_thread {
871
int exited;
872
} discovery_thread;
873
862
-// ----------------------------------------------------------------------------
874
+// ---------------------------------------------------------------------------------------------
875
+
876
+static inline int matches_enabled_cgroup_paths(char *id) {
877
+ if (!cgroup_enable_new_cgroups_detected_at_runtime) {
878
+ return 0;
879
+ }
880
+ return simple_pattern_matches(enabled_cgroup_paths, id);
881
+}
882
+
883
+static inline int matches_enabled_cgroup_names(char *name) {
884
+ return simple_pattern_matches(enabled_cgroup_names, name);
885
+}
886
+
887
+static inline int matches_enabled_cgroup_renames(char *id) {
888
+ return simple_pattern_matches(enabled_cgroup_renames, id);
889
+}
890
+
891
+static inline int matches_systemd_services_cgroups(char *id) {
892
+ return simple_pattern_matches(systemd_services_cgroups, id);
893
+}
894
+
895
+static inline int matches_search_cgroup_paths(const char *dir) {
896
+ return simple_pattern_matches(search_cgroup_paths, dir);
897
+}
898
+
899
+static inline int is_cgroup_systemd_service(struct cgroup *cg) {
900
+ return (cg->options & CGROUP_OPTIONS_SYSTEM_SLICE_SERVICE);
901
+}
902
+
903
+// ---------------------------------------------------------------------------------------------
904
+static int k8s_is_container(const char *id) {
905
+ // examples:
906
+ // https://github.com/netdata/netdata/blob/0fc101679dcd12f1cb8acdd07bb4c85d8e553e53/collectors/cgroups.plugin/cgroup-name.sh#L121-L147
907
+ const char *p = id;
908
+ const char *pp = NULL;
909
+ int i = 0;
910
+ size_t l = 3; // pod
911
+ while ((p = strstr(p, "pod"))) {
912
+ i++;
913
+ p += l;
914
+ pp = p;
915
+ }
916
+ return !(i < 2 || !pp || !(pp = strchr(pp, '/')) || !pp++ || !*pp);
917
+}
918
+
919
+#define TASK_COMM_LEN 16
920
+
921
+static int k8s_get_container_first_proc_comm(const char *id, char *comm) {
922
+ if (!k8s_is_container(id)) {
923
+ return 1;
924
+ }
925
+
926
+ static procfile *ff = NULL;
927
+
928
+ char filename[FILENAME_MAX + 1];
929
+ snprintfz(filename, FILENAME_MAX, "%s/%s/cgroup.procs", cgroup_cpuacct_base, id);
930
+
931
+ ff = procfile_reopen(ff, filename, NULL, PROCFILE_FLAG_DEFAULT);
932
+ if (unlikely(!ff)) {
933
+ debug(D_CGROUP, "CGROUP: k8s_is_pause_container(): cannot open file '%s'.", filename);
934
+ return 1;
935
+ }
936
+
937
+ ff = procfile_readall(ff);
938
+ if (unlikely(!ff)) {
939
+ debug(D_CGROUP, "CGROUP: k8s_is_pause_container(): cannot read file '%s'.", filename);
940
+ return 1;
941
+ }
942
+
943
+ unsigned long lines = procfile_lines(ff);
944
+ if (likely(lines < 2)) {
945
+ return 1;
946
+ }
947
+
948
+ char *pid = procfile_lineword(ff, 0, 0);
949
+ if (!pid || !*pid) {
950
+ return 1;
951
+ }
952
+
953
+ snprintfz(filename, FILENAME_MAX, "%s/proc/%s/comm", netdata_configured_host_prefix, pid);
954
+
955
+ ff = procfile_reopen(ff, filename, NULL, PROCFILE_FLAG_DEFAULT);
956
+ if (unlikely(!ff)) {
957
+ debug(D_CGROUP, "CGROUP: k8s_is_pause_container(): cannot open file '%s'.", filename);
958
+ return 1;
959
+ }
960
+
961
+ ff = procfile_readall(ff);
962
+ if (unlikely(!ff)) {
963
+ debug(D_CGROUP, "CGROUP: k8s_is_pause_container(): cannot read file '%s'.", filename);
964
+ return 1;
965
+ }
966
+
967
+ lines = procfile_lines(ff);
968
+ if (unlikely(lines != 2)) {
969
+ return 1;
970
+ }
971
+
972
+ char *proc_comm = procfile_lineword(ff, 0, 0);
973
+ if (!proc_comm || !*proc_comm) {
974
+ return 1;
975
+ }
976
+
977
+ strncpyz(comm, proc_comm, TASK_COMM_LEN);
978
+ return 0;
979
+}
980
+
981
+// ---------------------------------------------------------------------------------------------
982
983
static unsigned long long calc_delta(unsigned long long curr, unsigned long long prev) {
984
if (prev > curr) {
@@ -875,6 +994,15 @@ static unsigned long long calc_percentage(unsigned long long value, unsigned lon
994
return (calculated_number)value / (calculated_number)total * 100;
995
}
996
997
+static int calc_cgroup_depth(const char *id) {
998
+ int depth = 0;
999
+ const char *s;
1000
+ for (s = id; *s; s++) {
1001
+ depth += unlikely(*s == '/');
1002
+ }
1003
+ return depth;
1004
+}
1005
+
1006
// ----------------------------------------------------------------------------
1007
// read values from /sys
1008
@@ -1429,7 +1557,7 @@ memory_next:
1557
}
1558
}
1559
1432
-static inline void cgroup_read(struct cgroup *cg) {
1560
+static inline void read_cgroup(struct cgroup *cg) {
1561
debug(D_CGROUP, "reading metrics for cgroups '%s'", cg->id);
1562
if(!(cg->options & CGROUP_OPTIONS_IS_UNIFIED)) {
1563
cgroup_read_cpuacct_stat(&cg->cpuacct_stat);
@@ -1457,14 +1585,15 @@ static inline void cgroup_read(struct cgroup *cg) {
1585
}
1586
}
1587
1460
-static inline void read_all_cgroups(struct cgroup *root) {
1588
+static inline void read_all_discovered_cgroups(struct cgroup *root) {
1589
debug(D_CGROUP, "reading metrics for all cgroups");
1590
1591
struct cgroup *cg;
1464
-
1465
- for(cg = root; cg ; cg = cg->next)
1466
- if(cg->enabled && !cg->pending_renames)
1467
- cgroup_read(cg);
1592
+ for (cg = root; cg; cg = cg->next) {
1593
+ if (cg->enabled && !cg->pending_renames) {
1594
+ read_cgroup(cg);
1595
+ }
1596
+ }
1597
}
1598
1599
// ----------------------------------------------------------------------------
@@ -1582,8 +1711,7 @@ static inline void substitute_dots_in_id(char *s) {
1711
}
1712
}
1713
1585
-char *parse_k8s_data(struct label **labels, char *data)
1586
-{
1714
+char *k8s_parse_resolved_name(struct label **labels, char *data) {
1715
char *name = mystrsep(&data, " ");
1716
1717
if (!data) {
@@ -1611,181 +1739,6 @@ char *parse_k8s_data(struct label **labels, char *data)
1739
return name;
1740
}
1741
1614
-static inline void cgroup_get_chart_name(struct cgroup *cg) {
1615
- debug(D_CGROUP, "looking for the name of cgroup '%s' with chart id '%s' and title '%s'", cg->id, cg->chart_id, cg->chart_title);
1616
-
1617
- pid_t cgroup_pid;
1618
- // TODO: use cg->id when the renaming script is fixed
1619
- debug(D_CGROUP, "executing command %s \"%s\" for cgroup '%s'", cgroups_rename_script, cg->intermediate_id, cg->chart_id);
1620
- FILE *fp;
1621
- (void)mypopen_raw_default_flags_and_environment(&cgroup_pid, &fp, cgroups_rename_script, cg->intermediate_id);
1622
- if(fp) {
1623
- // debug(D_CGROUP, "reading from command '%s' for cgroup '%s'", command, cg->id);
1624
- char buffer[CGROUP_CHARTID_LINE_MAX + 1];
1625
- char *s = fgets(buffer, CGROUP_CHARTID_LINE_MAX, fp);
1626
- // debug(D_CGROUP, "closing command for cgroup '%s'", cg->id);
1627
- int name_error = mypclose(fp, cgroup_pid);
1628
- // debug(D_CGROUP, "closed command for cgroup '%s'", cg->id);
1629
-
1630
- if(s && *s && *s != '\n') {
1631
- debug(D_CGROUP, "cgroup '%s' should be renamed to '%s'", cg->chart_id, s);
1632
-
1633
- s = trim(s);
1634
- if (s) {
1635
- if(likely(name_error==0))
1636
- cg->pending_renames = 0;
1637
- else if (unlikely(name_error==3)) {
1638
- debug(D_CGROUP, "cgroup '%s' disabled based due to rename command output", cg->chart_id);
1639
- cg->enabled = 0;
1640
- }
1641
-
1642
- if (likely(cg->pending_renames < 2)) {
1643
- char *name = s;
1644
-
1645
- if (!strncmp(s, "k8s_", 4)) {
1646
- free_label_list(cg->chart_labels);
1647
- name = parse_k8s_data(&cg->chart_labels, s);
1648
- }
1649
-
1650
- freez(cg->chart_title);
1651
- cg->chart_title = cgroup_title_strdupz(name);
1652
-
1653
- freez(cg->chart_id);
1654
- cg->chart_id = cgroup_chart_id_strdupz(name);
1655
- substitute_dots_in_id(cg->chart_id);
1656
- cg->hash_chart = simple_hash(cg->chart_id);
1657
- }
1658
- }
1659
- }
1660
- }
1661
- else
1662
- error("CGROUP: cannot popen(%s \"%s\", \"r\").", cgroups_rename_script, cg->intermediate_id);
1663
-}
1664
-
1665
-static inline struct cgroup *cgroup_add(const char *id) {
1666
- if(!id || !*id) id = "/";
1667
- debug(D_CGROUP, "adding to list, cgroup with id '%s'", id);
1668
-
1669
- if(cgroup_root_count >= cgroup_root_max) {
1670
- info("CGROUP: maximum number of cgroups reached (%d). Not adding cgroup '%s'", cgroup_root_count, id);
1671
- return NULL;
1672
- }
1673
-
1674
- int def = simple_pattern_matches(enabled_cgroup_patterns, id)?cgroup_enable_new_cgroups_detected_at_runtime:0;
1675
- struct cgroup *cg = callocz(1, sizeof(struct cgroup));
1676
-
1677
- cg->id = strdupz(id);
1678
- cg->hash = simple_hash(cg->id);
1679
-
1680
- cg->chart_title = cgroup_title_strdupz(id);
1681
-
1682
- cg->intermediate_id = cgroup_chart_id_strdupz(id);
1683
-
1684
- cg->chart_id = cgroup_chart_id_strdupz(id);
1685
- substitute_dots_in_id(cg->chart_id);
1686
- cg->hash_chart = simple_hash(cg->chart_id);
1687
-
1688
- if(cgroup_use_unified_cgroups) cg->options |= CGROUP_OPTIONS_IS_UNIFIED;
1689
-
1690
- if(!discovered_cgroup_root)
1691
- discovered_cgroup_root = cg;
1692
- else {
1693
- // append it
1694
- struct cgroup *e;
1695
- for(e = discovered_cgroup_root; e->discovered_next ;e = e->discovered_next) ;
1696
- e->discovered_next = cg;
1697
- }
1698
-
1699
- cgroup_root_count++;
1700
-
1701
- // fix the chart_id and title by calling the external script
1702
- if(simple_pattern_matches(enabled_cgroup_renames, cg->id)) {
1703
-
1704
- cg->pending_renames = 2;
1705
- cgroup_get_chart_name(cg);
1706
-
1707
- debug(D_CGROUP, "cgroup '%s' renamed to '%s' (title: '%s')", cg->id, cg->chart_id, cg->chart_title);
1708
- }
1709
- else
1710
- debug(D_CGROUP, "cgroup '%s' will not be renamed - it matches the list of disabled cgroup renames (will be shown as '%s')", cg->id, cg->chart_id);
1711
-
1712
- int user_configurable = 1;
1713
-
1714
- // check if this cgroup should be a systemd service
1715
- if(cgroup_enable_systemd_services) {
1716
- if(simple_pattern_matches(systemd_services_cgroups, cg->id) ||
1717
- simple_pattern_matches(systemd_services_cgroups, cg->chart_id)) {
1718
- debug(D_CGROUP, "cgroup '%s' with chart id '%s' (title: '%s') matches systemd services cgroups", cg->id, cg->chart_id, cg->chart_title);
1719
-
1720
- char buffer[CGROUP_CHARTID_LINE_MAX + 1];
1721
- cg->options |= CGROUP_OPTIONS_SYSTEM_SLICE_SERVICE;
1722
-
1723
- strncpy(buffer, cg->id, CGROUP_CHARTID_LINE_MAX);
1724
- char *s = buffer;
1725
-
1726
- // skip to the last slash
1727
- size_t len = strlen(s);
1728
- while(len--) if(unlikely(s[len] == '/')) break;
1729
- if(len) s = &s[len + 1];
1730
-
1731
- // remove extension
1732
- len = strlen(s);
1733
- while(len--) if(unlikely(s[len] == '.')) break;
1734
- if(len) s[len] = '\0';
1735
-
1736
- freez(cg->chart_title);
1737
- cg->chart_title = cgroup_title_strdupz(s);
1738
-
1739
- cg->enabled = 1;
1740
- user_configurable = 0;
1741
-
1742
- debug(D_CGROUP, "cgroup '%s' renamed to '%s' (title: '%s')", cg->id, cg->chart_id, cg->chart_title);
1743
- }
1744
- else
1745
- debug(D_CGROUP, "cgroup '%s' with chart id '%s' (title: '%s') does not match systemd services groups", cg->id, cg->chart_id, cg->chart_title);
1746
- }
1747
-
1748
- if(user_configurable) {
1749
- // allow the user to enable/disable this individually
1750
- char option[FILENAME_MAX + 1];
1751
- snprintfz(option, FILENAME_MAX, "enable cgroup %s", cg->chart_title);
1752
- cg->enabled = (char) config_get_boolean("plugin:cgroups", option, def);
1753
- }
1754
-
1755
- // detect duplicate cgroups
1756
- if(cg->enabled) {
1757
- struct cgroup *t;
1758
- for (t = discovered_cgroup_root; t; t = t->discovered_next) {
1759
- if (t != cg && t->enabled && t->hash_chart == cg->hash_chart && !strcmp(t->chart_id, cg->chart_id)) {
1760
- // TODO: use it after refactoring if system.slice might be scanned before init.scope/system.slice
1761
- //
1762
- // if (!strncmp(t->id, "/system.slice/", 14) && !strncmp(cg->id, "/init.scope/system.slice/", 25)) {
1763
- // error("CGROUP: chart id '%s' already exists with id '%s' and is enabled. Swapping them by enabling cgroup with id '%s' and disabling cgroup with id '%s'.",
1764
- // cg->chart_id, t->id, cg->id, t->id);
1765
- // t->enabled = 0;
1766
- // t->options |= CGROUP_OPTIONS_DISABLED_DUPLICATE;
1767
- // }
1768
- // else {}
1769
- //
1770
- // https://github.com/netdata/netdata/issues/797#issuecomment-241248884
1771
- error("CGROUP: chart id '%s' already exists with id '%s' and is enabled and available. Disabling cgroup with id '%s'.",
1772
- cg->chart_id, t->id, cg->id);
1773
- cg->enabled = 0;
1774
- cg->options |= CGROUP_OPTIONS_DISABLED_DUPLICATE;
1775
-
1776
- break;
1777
- }
1778
- }
1779
- }
1780
-
1781
- if(cg->enabled && !cg->pending_renames && !(cg->options & CGROUP_OPTIONS_SYSTEM_SLICE_SERVICE))
1782
- read_cgroup_network_interfaces(cg);
1783
-
1784
- debug(D_CGROUP, "ADDED CGROUP: '%s' with chart id '%s' and title '%s' as %s (default was %s)", cg->id, cg->chart_id, cg->chart_title, (cg->enabled)?"enabled":"disabled", (def)?"enabled":"disabled");
1785
-
1786
- return cg;
1787
-}
1788
-
1742
static inline void free_pressure(struct pressure *res) {
1743
if (res->some.st) rrdset_is_obsolete(res->some.st);
1744
if (res->full.st) rrdset_is_obsolete(res->full.st);
@@ -1862,71 +1815,201 @@ static inline void cgroup_free(struct cgroup *cg) {
1815
cgroup_root_count--;
1816
}
1817
1865
-// find if a given cgroup exists
1866
-static inline struct cgroup *cgroup_find(const char *id) {
1867
- debug(D_CGROUP, "searching for cgroup '%s'", id);
1818
+// ----------------------------------------------------------------------------
1819
1869
- uint32_t hash = simple_hash(id);
1820
+static inline void discovery_rename_cgroup(struct cgroup *cg) {
1821
+ if (!cg->pending_renames) {
1822
+ return;
1823
+ }
1824
+ cg->pending_renames--;
1825
1871
- struct cgroup *cg;
1872
- for(cg = discovered_cgroup_root; cg ; cg = cg->discovered_next) {
1873
- if(hash == cg->hash && strcmp(id, cg->id) == 0)
1826
+ debug(D_CGROUP, "looking for the name of cgroup '%s' with chart id '%s' and title '%s'", cg->id, cg->chart_id, cg->chart_title);
1827
+ debug(D_CGROUP, "executing command %s \"%s\" for cgroup '%s'", cgroups_rename_script, cg->intermediate_id, cg->chart_id);
1828
+ pid_t cgroup_pid;
1829
+
1830
+ FILE *fp;
1831
+ (void)mypopen_raw_default_flags_and_environment(&cgroup_pid, &fp, cgroups_rename_script, cg->id, cg->intermediate_id);
1832
+ if (!fp) {
1833
+ error("CGROUP: cannot popen(%s \"%s\", \"r\").", cgroups_rename_script, cg->intermediate_id);
1834
+ cg->pending_renames = 0;
1835
+ cg->processed = 1;
1836
+ return;
1837
+ }
1838
+
1839
+ char buffer[CGROUP_CHARTID_LINE_MAX + 1];
1840
+ char *new_name = fgets(buffer, CGROUP_CHARTID_LINE_MAX, fp);
1841
+ int exit_code = mypclose(fp, cgroup_pid);
1842
+
1843
+ switch (exit_code) {
1844
+ case 0:
1845
+ cg->pending_renames = 0;
1846
+ break;
1847
+ case 3:
1848
+ cg->pending_renames = 0;
1849
+ cg->processed = 1;
1850
break;
1851
+ default:
1852
+ if (!cg->pending_renames && is_inside_k8s) {
1853
+ cg->processed = 1;
1854
+ }
1855
}
1856
1877
- debug(D_CGROUP, "cgroup '%s' %s in memory", id, (cg)?"found":"not found");
1878
- return cg;
1857
+ if (cg->pending_renames || cg->processed) {
1858
+ return;
1859
+ }
1860
+ if (!(new_name && *new_name && *new_name != '\n')) {
1861
+ return;
1862
+ }
1863
+ new_name = trim(new_name);
1864
+ if (!(new_name)) {
1865
+ return;
1866
+ }
1867
+ char *name = new_name;
1868
+ if (!strncmp(new_name, "k8s_", 4)) {
1869
+ free_label_list(cg->chart_labels);
1870
+ name = k8s_parse_resolved_name(&cg->chart_labels, new_name);
1871
+ }
1872
+ freez(cg->chart_title);
1873
+ cg->chart_title = cgroup_title_strdupz(name);
1874
+ freez(cg->chart_id);
1875
+ cg->chart_id = cgroup_chart_id_strdupz(name);
1876
+ substitute_dots_in_id(cg->chart_id);
1877
+ cg->hash_chart = simple_hash(cg->chart_id);
1878
}
1879
1881
-// ----------------------------------------------------------------------------
1882
-// detect running cgroups
1880
+static void is_cgroup_procs_exist(netdata_ebpf_cgroup_shm_body_t *out, char *id) {
1881
+ struct stat buf;
1882
1884
-// callback for find_file_in_subdirs()
1885
-static inline void found_subdir_in_dir(const char *dir) {
1886
- debug(D_CGROUP, "examining cgroup dir '%s'", dir);
1883
+ snprintfz(out->path, FILENAME_MAX, "%s%s/cgroup.procs", cgroup_cpuset_base, id);
1884
+ if (likely(stat(out->path, &buf) == 0)) {
1885
+ return;
1886
+ }
1887
1888
- struct cgroup *cg = cgroup_find(dir);
1889
- if(!cg) {
1890
- if(*dir && cgroup_max_depth > 0) {
1891
- int depth = 0;
1892
- const char *s;
1888
+ snprintfz(out->path, FILENAME_MAX, "%s%s/cgroup.procs", cgroup_blkio_base, id);
1889
+ if (likely(stat(out->path, &buf) == 0)) {
1890
+ return;
1891
+ }
1892
1894
- for(s = dir; *s ;s++)
1895
- if(unlikely(*s == '/'))
1896
- depth++;
1893
+ snprintfz(out->path, FILENAME_MAX, "%s%s/cgroup.procs", cgroup_memory_base, id);
1894
+ if (likely(stat(out->path, &buf) == 0)) {
1895
+ return;
1896
+ }
1897
1898
- if(depth > cgroup_max_depth) {
1899
- info("CGROUP: '%s' is too deep (%d, while max is %d)", dir, depth, cgroup_max_depth);
1900
- return;
1901
- }
1898
+ snprintfz(out->path, FILENAME_MAX, "%s%s/cgroup.procs", cgroup_devices_base, id);
1899
+ if (likely(stat(out->path, &buf) == 0)) {
1900
+ return;
1901
+ }
1902
+
1903
+ out->path[0] = '\0';
1904
+ out->enabled = 0;
1905
+}
1906
+
1907
+static inline void convert_cgroup_to_systemd_service(struct cgroup *cg) {
1908
+ char buffer[CGROUP_CHARTID_LINE_MAX];
1909
+ cg->options |= CGROUP_OPTIONS_SYSTEM_SLICE_SERVICE;
1910
+ strncpyz(buffer, cg->id, CGROUP_CHARTID_LINE_MAX);
1911
+ char *s = buffer;
1912
+
1913
+ // skip to the last slash
1914
+ size_t len = strlen(s);
1915
+ while (len--) {
1916
+ if (unlikely(s[len] == '/')) {
1917
+ break;
1918
}
1903
- // debug(D_CGROUP, "will add dir '%s' as cgroup", dir);
1904
- cg = cgroup_add(dir);
1919
+ }
1920
+ if (len) {
1921
+ s = &s[len + 1];
1922
}
1923
1907
- if(cg) {
1908
- // delay renaming of the cgroup and looking for network interfaces to deal with the docker lag when starting the container
1909
- if(unlikely(cg->pending_renames == 1)) {
1910
- // fix the chart_id and title by calling the external script
1911
- if(simple_pattern_matches(enabled_cgroup_renames, cg->id)) {
1924
+ // remove extension
1925
+ len = strlen(s);
1926
+ while (len--) {
1927
+ if (unlikely(s[len] == '.')) {
1928
+ break;
1929
+ }
1930
+ }
1931
+ if (len) {
1932
+ s[len] = '\0';
1933
+ }
1934
1913
- cgroup_get_chart_name(cg);
1914
- cg->pending_renames = 0;
1935
+ freez(cg->chart_title);
1936
+ cg->chart_title = cgroup_title_strdupz(s);
1937
+}
1938
1916
- if(cg->enabled && !(cg->options & CGROUP_OPTIONS_SYSTEM_SLICE_SERVICE))
1917
- read_cgroup_network_interfaces(cg);
1939
+static inline struct cgroup *discovery_cgroup_add(const char *id) {
1940
+ debug(D_CGROUP, "adding to list, cgroup with id '%s'", id);
1941
1919
- debug(D_CGROUP, "cgroup '%s' renamed to '%s' (title: '%s')", cg->id, cg->chart_id, cg->chart_title);
1920
- }
1921
- else
1922
- debug(D_CGROUP, "cgroup '%s' will not be renamed - it matches the list of disabled cgroup renames (will be shown as '%s')", cg->id, cg->chart_id);
1942
+ struct cgroup *cg = callocz(1, sizeof(struct cgroup));
1943
+ cg->id = strdupz(id);
1944
+ cg->hash = simple_hash(cg->id);
1945
+ cg->chart_title = cgroup_title_strdupz(id);
1946
+ cg->intermediate_id = cgroup_chart_id_strdupz(id);
1947
+ cg->chart_id = cgroup_chart_id_strdupz(id);
1948
+ substitute_dots_in_id(cg->chart_id);
1949
+ cg->hash_chart = simple_hash(cg->chart_id);
1950
+ if (cgroup_use_unified_cgroups) {
1951
+ cg->options |= CGROUP_OPTIONS_IS_UNIFIED;
1952
+ }
1953
+
1954
+ if (!discovered_cgroup_root)
1955
+ discovered_cgroup_root = cg;
1956
+ else {
1957
+ struct cgroup *t;
1958
+ for (t = discovered_cgroup_root; t->discovered_next; t = t->discovered_next) {
1959
}
1960
+ t->discovered_next = cg;
1961
+ }
1962
+
1963
+ return cg;
1964
+}
1965
+
1966
+static inline struct cgroup *discovery_cgroup_find(const char *id) {
1967
+ debug(D_CGROUP, "searching for cgroup '%s'", id);
1968
1969
+ uint32_t hash = simple_hash(id);
1970
+
1971
+ struct cgroup *cg;
1972
+ for(cg = discovered_cgroup_root; cg ; cg = cg->discovered_next) {
1973
+ if(hash == cg->hash && strcmp(id, cg->id) == 0)
1974
+ break;
1975
+ }
1976
+
1977
+ debug(D_CGROUP, "cgroup '%s' %s in memory", id, (cg)?"found":"not found");
1978
+ return cg;
1979
+}
1980
+
1981
+static inline void discovery_find_cgroup_in_dir_callback(const char *dir) {
1982
+ if (!dir || !*dir) {
1983
+ dir = "/";
1984
+ }
1985
+ debug(D_CGROUP, "examining cgroup dir '%s'", dir);
1986
+
1987
+ struct cgroup *cg = discovery_cgroup_find(dir);
1988
+ if (cg) {
1989
cg->available = 1;
1990
+ return;
1991
}
1992
+
1993
+ if (cgroup_root_count >= cgroup_root_max) {
1994
+ info("CGROUP: maximum number of cgroups reached (%d). Not adding cgroup '%s'", cgroup_root_count, dir);
1995
+ return;
1996
+ }
1997
+
1998
+ if (cgroup_max_depth > 0) {
1999
+ int depth = calc_cgroup_depth(dir);
2000
+ if (depth > cgroup_max_depth) {
2001
+ info("CGROUP: '%s' is too deep (%d, while max is %d)", dir, depth, cgroup_max_depth);
2002
+ return;
2003
+ }
2004
+ }
2005
+
2006
+ cg = discovery_cgroup_add(dir);
2007
+ cg->available = 1;
2008
+ cg->first_time_seen = 1;
2009
+ cgroup_root_count++;
2010
}
2011
1929
-static inline int find_dir_in_subdirs(const char *base, const char *this, void (*callback)(const char *)) {
2012
+static inline int discovery_find_dir_in_subdirs(const char *base, const char *this, void (*callback)(const char *)) {
2013
if(!this) this = base;
2014
debug(D_CGROUP, "searching for directories in '%s' (base '%s')", this?this:"", base);
2015
@@ -1962,7 +2045,7 @@ static inline int find_dir_in_subdirs(const char *base, const char *this, void (
2045
if(*r == '\0') r = "/";
2046
2047
// do not decent in directories we are not interested
1965
- enabled = simple_pattern_matches(enabled_cgroup_paths, r);
2048
+ enabled = matches_search_cgroup_paths(r);
2049
}
2050
2051
if(enabled) {
@@ -1970,7 +2053,7 @@ static inline int find_dir_in_subdirs(const char *base, const char *this, void (
2053
strcpy(s, this);
2054
strcat(s, "/");
2055
strcat(s, de->d_name);
1973
- int ret2 = find_dir_in_subdirs(base, s, callback);
2056
+ int ret2 = discovery_find_dir_in_subdirs(base, s, callback);
2057
if(ret2 > 0) ret += ret2;
2058
freez(s);
2059
}
@@ -1981,28 +2064,19 @@ static inline int find_dir_in_subdirs(const char *base, const char *this, void (
2064
return ret;
2065
}
2066
1984
-static inline void mark_all_cgroups_as_not_available() {
2067
+static inline void discovery_mark_all_cgroups_as_unavailable() {
2068
debug(D_CGROUP, "marking all cgroups as not available");
1986
-
2069
struct cgroup *cg;
1988
-
1989
- // mark all as not available
1990
- for(cg = discovered_cgroup_root; cg ; cg = cg->discovered_next) {
2070
+ for (cg = discovered_cgroup_root; cg; cg = cg->discovered_next) {
2071
cg->available = 0;
2072
}
2073
}
2074
1995
-static inline void update_filenames()
1996
-{
2075
+static inline void discovery_update_filenames() {
2076
struct cgroup *cg;
2077
struct stat buf;
2078
for(cg = discovered_cgroup_root; cg ; cg = cg->discovered_next) {
2000
- // fprintf(stderr, " >>> CGROUP '%s' (%u - %s) with name '%s'\n", cg->id, cg->hash, cg->available?"available":"stopped", cg->name);
2001
-
2002
- if(unlikely(cg->pending_renames))
2003
- cg->pending_renames--;
2004
-
2005
- if(unlikely(!cg->available || cg->pending_renames))
2079
+ if(unlikely(!cg->available || !cg->enabled || cg->pending_renames))
2080
continue;
2081
2082
debug(D_CGROUP, "checking paths for cgroup '%s'", cg->id);
@@ -2028,7 +2102,7 @@ static inline void update_filenames()
2102
debug(D_CGROUP, "cpuacct.stat file for cgroup '%s': '%s' does not exist.", cg->id, filename);
2103
}
2104
2031
- if(unlikely(cgroup_enable_cpuacct_usage && !cg->cpuacct_usage.filename && !(cg->options & CGROUP_OPTIONS_SYSTEM_SLICE_SERVICE))) {
2105
+ if(unlikely(cgroup_enable_cpuacct_usage && !cg->cpuacct_usage.filename && !is_cgroup_systemd_service(cg))) {
2106
snprintfz(filename, FILENAME_MAX, "%s%s/cpuacct.usage_percpu", cgroup_cpuacct_base, cg->id);
2107
if(likely(stat(filename, &buf) != -1)) {
2108
cg->cpuacct_usage.filename = strdupz(filename);
@@ -2038,7 +2112,7 @@ static inline void update_filenames()
2112
else
2113
debug(D_CGROUP, "cpuacct.usage_percpu file for cgroup '%s': '%s' does not exist.", cg->id, filename);
2114
}
2041
- if(unlikely(cgroup_enable_cpuacct_cpu_throttling && !cg->cpuacct_cpu_throttling.filename && !(cg->options & CGROUP_OPTIONS_SYSTEM_SLICE_SERVICE))) {
2115
+ if(unlikely(cgroup_enable_cpuacct_cpu_throttling && !cg->cpuacct_cpu_throttling.filename && !is_cgroup_systemd_service(cg))) {
2116
snprintfz(filename, FILENAME_MAX, "%s%s/cpu.stat", cgroup_cpuacct_base, cg->id);
2117
if(likely(stat(filename, &buf) != -1)) {
2118
cg->cpuacct_cpu_throttling.filename = strdupz(filename);
@@ -2050,7 +2124,7 @@ static inline void update_filenames()
2124
}
2125
if (unlikely(
2126
cgroup_enable_cpuacct_cpu_shares && !cg->cpuacct_cpu_shares.filename &&
2053
- !(cg->options & CGROUP_OPTIONS_SYSTEM_SLICE_SERVICE))) {
2127
+ !is_cgroup_systemd_service(cg))) {
2128
snprintfz(filename, FILENAME_MAX, "%s%s/cpu.shares", cgroup_cpuacct_base, cg->id);
2129
if (likely(stat(filename, &buf) != -1)) {
2130
cg->cpuacct_cpu_shares.filename = strdupz(filename);
@@ -2061,7 +2135,7 @@ static inline void update_filenames()
2135
debug(D_CGROUP, "cpu.shares file for cgroup '%s': '%s' does not exist.", cg->id, filename);
2136
}
2137
2064
- if(unlikely((cgroup_enable_detailed_memory || cgroup_used_memory) && !cg->memory.filename_detailed && (cgroup_used_memory || cgroup_enable_systemd_services_detailed_memory || !(cg->options & CGROUP_OPTIONS_SYSTEM_SLICE_SERVICE)))) {
2138
+ if(unlikely((cgroup_enable_detailed_memory || cgroup_used_memory) && !cg->memory.filename_detailed && (cgroup_used_memory || cgroup_enable_systemd_services_detailed_memory || !is_cgroup_systemd_service(cg)))) {
2139
snprintfz(filename, FILENAME_MAX, "%s%s/memory.stat", cgroup_memory_base, cg->id);
2140
if(likely(stat(filename, &buf) != -1)) {
2141
cg->memory.filename_detailed = strdupz(filename);
@@ -2270,7 +2344,7 @@ static inline void update_filenames()
2344
debug(D_CGROUP, "cpu.weight file for cgroup '%s': '%s' does not exist.", cg->id, filename);
2345
}
2346
2273
- if(unlikely((cgroup_enable_detailed_memory || cgroup_used_memory) && !cg->memory.filename_detailed && (cgroup_used_memory || cgroup_enable_systemd_services_detailed_memory || !(cg->options & CGROUP_OPTIONS_SYSTEM_SLICE_SERVICE)))) {
2347
+ if(unlikely((cgroup_enable_detailed_memory || cgroup_used_memory) && !cg->memory.filename_detailed && (cgroup_used_memory || cgroup_enable_systemd_services_detailed_memory || !is_cgroup_systemd_service(cg)))) {
2348
snprintfz(filename, FILENAME_MAX, "%s%s/memory.stat", cgroup_unified_base, cg->id);
2349
if(likely(stat(filename, &buf) != -1)) {
2350
cg->memory.filename_detailed = strdupz(filename);
@@ -2346,7 +2420,7 @@ static inline void update_filenames()
2420
}
2421
}
2422
2349
-static inline void cleanup_all_cgroups() {
2423
+static inline void discovery_cleanup_all_cgroups() {
2424
struct cgroup *cg = discovered_cgroup_root, *last = NULL;
2425
2426
for(; cg ;) {
@@ -2369,9 +2443,6 @@ static inline void cleanup_all_cgroups() {
2443
else
2444
last->discovered_next = cg->discovered_next;
2445
2372
- char option[FILENAME_MAX + 1];
2373
- snprintfz(option, FILENAME_MAX, "enable cgroup %s", cg->chart_title);
2374
- config_section_option_destroy("plugin:cgroups", option);
2446
cgroup_free(cg);
2447
2448
if(!last)
@@ -2386,49 +2457,19 @@ static inline void cleanup_all_cgroups() {
2457
}
2458
}
2459
2389
-static inline void copy_discovered_cgroups()
2390
-{
2460
+static inline void discovery_copy_discovered_cgroups_to_reader() {
2461
debug(D_CGROUP, "copy discovered cgroups to the main group list");
2462
2463
struct cgroup *cg;
2464
2395
- for(cg = discovered_cgroup_root; cg ; cg = cg->discovered_next) {
2465
+ for (cg = discovered_cgroup_root; cg; cg = cg->discovered_next) {
2466
cg->next = cg->discovered_next;
2467
}
2468
2469
cgroup_root = discovered_cgroup_root;
2470
}
2471
2402
-static void is_there_cgroup_procs(netdata_ebpf_cgroup_shm_body_t *out, char *id)
2403
-{
2404
- struct stat buf;
2405
-
2406
- snprintfz(out->path, FILENAME_MAX, "%s%s/cgroup.procs", cgroup_cpuset_base, id);
2407
- if (likely(stat(out->path, &buf) == 0)) {
2408
- return;
2409
- }
2410
-
2411
- snprintfz(out->path, FILENAME_MAX, "%s%s/cgroup.procs", cgroup_blkio_base, id);
2412
- if (likely(stat(out->path, &buf) == 0)) {
2413
- return;
2414
- }
2415
-
2416
- snprintfz(out->path, FILENAME_MAX, "%s%s/cgroup.procs", cgroup_memory_base, id);
2417
- if (likely(stat(out->path, &buf) == 0)) {
2418
- return;
2419
- }
2420
-
2421
- snprintfz(out->path, FILENAME_MAX, "%s%s/cgroup.procs", cgroup_devices_base, id);
2422
- if (likely(stat(out->path, &buf) == 0)) {
2423
- return;
2424
- }
2425
-
2426
- out->path[0] = '\0';
2427
- out->enabled = 0;
2428
-}
2429
-
2430
-static inline void share_cgroups()
2431
-{
2472
+static inline void discovery_share_cgroups_with_ebpf() {
2473
struct cgroup *cg;
2474
int count;
2475
struct stat buf;
@@ -2438,9 +2479,9 @@ static inline void share_cgroups()
2479
}
2480
sem_wait(shm_mutex_cgroup_ebpf);
2481
2441
- for (cg = cgroup_root, count = 0; cg ; cg = cg->next, count++) {
2482
+ for (cg = cgroup_root, count = 0; cg; cg = cg->next, count++) {
2483
netdata_ebpf_cgroup_shm_body_t *ptr = &shm_cgroup_ebpf.body[count];
2443
- char *prefix = (cg->options & CGROUP_OPTIONS_SYSTEM_SLICE_SERVICE) ? "" : "cgroup_";
2484
+ char *prefix = (is_cgroup_systemd_service(cg)) ? "" : "cgroup_";
2485
snprintfz(ptr->name, CGROUP_EBPF_NAME_SHARED_LENGTH - 1, "%s%s", prefix, cg->chart_title);
2486
ptr->hash = simple_hash(ptr->name);
2487
ptr->options = cg->options;
@@ -2452,7 +2493,7 @@ static inline void share_cgroups()
2493
ptr->enabled = 0;
2494
}
2495
} else {
2455
- is_there_cgroup_procs(ptr, cg->id);
2496
+ is_cgroup_procs_exist(ptr, cg->id);
2497
}
2498
2499
debug(D_CGROUP, "cgroup shared: NAME=%s, ENABLED=%d", ptr->name, ptr->enabled);
@@ -2462,63 +2503,173 @@ static inline void share_cgroups()
2503
sem_post(shm_mutex_cgroup_ebpf);
2504
}
2505
2465
-static inline void find_all_cgroups() {
2466
- debug(D_CGROUP, "searching for cgroups");
2506
+static inline void discovery_find_all_cgroups_v1() {
2507
+ if (cgroup_enable_cpuacct_stat || cgroup_enable_cpuacct_usage) {
2508
+ if (discovery_find_dir_in_subdirs(cgroup_cpuacct_base, NULL, discovery_find_cgroup_in_dir_callback) == -1) {
2509
+ cgroup_enable_cpuacct_stat = cgroup_enable_cpuacct_usage = CONFIG_BOOLEAN_NO;
2510
+ error("CGROUP: disabled cpu statistics.");
2511
+ }
2512
+ }
2513
2468
- mark_all_cgroups_as_not_available();
2469
- if(!cgroup_use_unified_cgroups) {
2470
- if(cgroup_enable_cpuacct_stat || cgroup_enable_cpuacct_usage) {
2471
- if(find_dir_in_subdirs(cgroup_cpuacct_base, NULL, found_subdir_in_dir) == -1) {
2472
- cgroup_enable_cpuacct_stat =
2473
- cgroup_enable_cpuacct_usage = CONFIG_BOOLEAN_NO;
2474
- error("CGROUP: disabled cpu statistics.");
2475
- }
2514
+ if (cgroup_enable_blkio_io || cgroup_enable_blkio_ops || cgroup_enable_blkio_throttle_io ||
2515
+ cgroup_enable_blkio_throttle_ops || cgroup_enable_blkio_merged_ops || cgroup_enable_blkio_queued_ops) {
2516
+ if (discovery_find_dir_in_subdirs(cgroup_blkio_base, NULL, discovery_find_cgroup_in_dir_callback) == -1) {
2517
+ cgroup_enable_blkio_io = cgroup_enable_blkio_ops = cgroup_enable_blkio_throttle_io =
2518
+ cgroup_enable_blkio_throttle_ops = cgroup_enable_blkio_merged_ops = cgroup_enable_blkio_queued_ops =
2519
+ CONFIG_BOOLEAN_NO;
2520
+ error("CGROUP: disabled blkio statistics.");
2521
}
2522
+ }
2523
2478
- if(cgroup_enable_blkio_io || cgroup_enable_blkio_ops || cgroup_enable_blkio_throttle_io || cgroup_enable_blkio_throttle_ops || cgroup_enable_blkio_merged_ops || cgroup_enable_blkio_queued_ops) {
2479
- if(find_dir_in_subdirs(cgroup_blkio_base, NULL, found_subdir_in_dir) == -1) {
2480
- cgroup_enable_blkio_io =
2481
- cgroup_enable_blkio_ops =
2482
- cgroup_enable_blkio_throttle_io =
2483
- cgroup_enable_blkio_throttle_ops =
2484
- cgroup_enable_blkio_merged_ops =
2485
- cgroup_enable_blkio_queued_ops = CONFIG_BOOLEAN_NO;
2486
- error("CGROUP: disabled blkio statistics.");
2487
- }
2524
+ if (cgroup_enable_memory || cgroup_enable_detailed_memory || cgroup_enable_swap || cgroup_enable_memory_failcnt) {
2525
+ if (discovery_find_dir_in_subdirs(cgroup_memory_base, NULL, discovery_find_cgroup_in_dir_callback) == -1) {
2526
+ cgroup_enable_memory = cgroup_enable_detailed_memory = cgroup_enable_swap = cgroup_enable_memory_failcnt =
2527
+ CONFIG_BOOLEAN_NO;
2528
+ error("CGROUP: disabled memory statistics.");
2529
}
2530
+ }
2531
2490
- if(cgroup_enable_memory || cgroup_enable_detailed_memory || cgroup_enable_swap || cgroup_enable_memory_failcnt) {
2491
- if(find_dir_in_subdirs(cgroup_memory_base, NULL, found_subdir_in_dir) == -1) {
2492
- cgroup_enable_memory =
2493
- cgroup_enable_detailed_memory =
2494
- cgroup_enable_swap =
2495
- cgroup_enable_memory_failcnt = CONFIG_BOOLEAN_NO;
2496
- error("CGROUP: disabled memory statistics.");
2497
- }
2532
+ if (cgroup_search_in_devices) {
2533
+ if (discovery_find_dir_in_subdirs(cgroup_devices_base, NULL, discovery_find_cgroup_in_dir_callback) == -1) {
2534
+ cgroup_search_in_devices = 0;
2535
+ error("CGROUP: disabled devices statistics.");
2536
}
2537
+ }
2538
+}
2539
2500
- if(cgroup_search_in_devices) {
2501
- if(find_dir_in_subdirs(cgroup_devices_base, NULL, found_subdir_in_dir) == -1) {
2502
- cgroup_search_in_devices = 0;
2503
- error("CGROUP: disabled devices statistics.");
2504
- }
2540
+static inline void discovery_find_all_cgroups_v2() {
2541
+ if (discovery_find_dir_in_subdirs(cgroup_unified_base, NULL, discovery_find_cgroup_in_dir_callback) == -1) {
2542
+ cgroup_unified_exist = CONFIG_BOOLEAN_NO;
2543
+ error("CGROUP: disabled unified cgroups statistics.");
2544
+ }
2545
+}
2546
+
2547
+static inline void discovery_process_first_time_seen_cgroup(struct cgroup *cg) {
2548
+ if (!cg->first_time_seen) {
2549
+ return;
2550
+ }
2551
+ cg->first_time_seen = 0;
2552
+
2553
+ char comm[TASK_COMM_LEN];
2554
+
2555
+ if (is_inside_k8s && !k8s_get_container_first_proc_comm(cg->id, comm)) {
2556
+ // container initialization may take some time when CPU % is high
2557
+ // TODO: not sure run-level 2 is enough (just came across this problem on an AWS K8s cluster)
2558
+ if (!strcmp(comm, "runc:[2:INIT]")) {
2559
+ cg->first_time_seen = 1;
2560
+ return;
2561
+ }
2562
+ if (!strcmp(comm, "pause")) {
2563
+ // a container that holds the network namespace for the pod
2564
+ // we don't need to collect its metrics
2565
+ cg->processed = 1;
2566
+ return;
2567
}
2568
}
2507
- else {
2508
- if (find_dir_in_subdirs(cgroup_unified_base, NULL, found_subdir_in_dir) == -1) {
2509
- cgroup_unified_exist = CONFIG_BOOLEAN_NO;
2510
- error("CGROUP: disabled unified cgroups statistics.");
2569
+
2570
+ if (cgroup_enable_systemd_services && matches_systemd_services_cgroups(cg->id)) {
2571
+ debug(D_CGROUP, "cgroup '%s' (name '%s') matches 'cgroups to match as systemd services'", cg->id, cg->chart_title);
2572
+ convert_cgroup_to_systemd_service(cg);
2573
+ return;
2574
+ }
2575
+
2576
+ if (matches_enabled_cgroup_renames(cg->id)) {
2577
+ debug(D_CGROUP, "cgroup '%s' (name '%s') matches 'run script to rename cgroups matching', will try to rename it", cg->id, cg->chart_title);
2578
+ if (is_inside_k8s && k8s_is_container(cg->id)) {
2579
+ // it may take up to a minute for the K8s API to return data for the container
2580
+ // tested on AWS K8s cluster with 100% CPU utilization
2581
+ cg->pending_renames = 9; // 1.5 minute
2582
+ } else {
2583
+ cg->pending_renames = 2;
2584
}
2585
}
2586
+}
2587
+
2588
+static int discovery_is_cgroup_duplicate(struct cgroup *cg) {
2589
+ // https://github.com/netdata/netdata/issues/797#issuecomment-241248884
2590
+ struct cgroup *c;
2591
+ for (c = discovered_cgroup_root; c; c = c->discovered_next) {
2592
+ if (c != cg && c->enabled && c->hash_chart == cg->hash_chart && !strcmp(c->chart_id, cg->chart_id)) {
2593
+ error("CGROUP: chart id '%s' already exists with id '%s' and is enabled and available. Disabling cgroup with id '%s'.", cg->chart_id, c->id, cg->id);
2594
+ return 1;
2595
+ }
2596
+ }
2597
+ return 0;
2598
+}
2599
+
2600
+static inline void discovery_process_cgroup(struct cgroup *cg) {
2601
+ if (!cg) {
2602
+ debug(D_CGROUP, "discovery_process_cgroup() received NULL");
2603
+ return;
2604
+ }
2605
+ if (!cg->available || cg->processed) {
2606
+ return;
2607
+ }
2608
+
2609
+ if (cg->first_time_seen) {
2610
+ discovery_process_first_time_seen_cgroup(cg);
2611
+ if (unlikely(cg->first_time_seen || cg->processed)) {
2612
+ return;
2613
+ }
2614
+ }
2615
+
2616
+ if (cg->pending_renames) {
2617
+ discovery_rename_cgroup(cg);
2618
+ if (unlikely(cg->pending_renames || cg->processed)) {
2619
+ return;
2620
+ }
2621
+ }
2622
+
2623
+ cg->processed = 1;
2624
+
2625
+ if (is_cgroup_systemd_service(cg)) {
2626
+ cg->enabled = 1;
2627
+ return;
2628
+ }
2629
+
2630
+ if (!(cg->enabled = matches_enabled_cgroup_names(cg->chart_title))) {
2631
+ debug(D_CGROUP, "cgroup '%s' (name '%s') disabled by 'enable by default cgroups names matching'", cg->id, cg->chart_title);
2632
+ return;
2633
+ }
2634
+
2635
+ if (!(cg->enabled = matches_enabled_cgroup_paths(cg->id))) {
2636
+ debug(D_CGROUP, "cgroup '%s' (name '%s') disabled by 'enable by default cgroups matching'", cg->id, cg->chart_title);
2637
+ return;
2638
+ }
2639
+
2640
+ if (discovery_is_cgroup_duplicate(cg)) {
2641
+ cg->enabled = 0;
2642
+ cg->options |= CGROUP_OPTIONS_DISABLED_DUPLICATE;
2643
+ return;
2644
+ }
2645
+
2646
+ read_cgroup_network_interfaces(cg);
2647
+}
2648
+
2649
+static inline void discovery_find_all_cgroups() {
2650
+ debug(D_CGROUP, "searching for cgroups");
2651
+
2652
+ discovery_mark_all_cgroups_as_unavailable();
2653
+
2654
+ if (!cgroup_use_unified_cgroups) {
2655
+ discovery_find_all_cgroups_v1();
2656
+ } else {
2657
+ discovery_find_all_cgroups_v2();
2658
+ }
2659
+
2660
+ struct cgroup *cg;
2661
+ for (cg = discovered_cgroup_root; cg; cg = cg->discovered_next) {
2662
+ discovery_process_cgroup(cg);
2663
+ }
2664
2514
- update_filenames();
2665
+ discovery_update_filenames();
2666
2667
uv_mutex_lock(&cgroup_root_mutex);
2517
- cleanup_all_cgroups();
2518
- copy_discovered_cgroups();
2668
+ discovery_cleanup_all_cgroups();
2669
+ discovery_copy_discovered_cgroups_to_reader();
2670
uv_mutex_unlock(&cgroup_root_mutex);
2671
2521
- share_cgroups();
2672
+ discovery_share_cgroups_with_ebpf();
2673
2674
debug(D_CGROUP, "done searching for cgroups");
2675
}
@@ -2537,7 +2688,7 @@ void cgroup_discovery_worker(void *ptr)
2688
if (unlikely(netdata_exit))
2689
break;
2690
2540
- find_all_cgroups();
2691
+ discovery_find_all_cgroups();
2692
}
2693
2694
discovery_thread.exited = 1;
@@ -3123,7 +3274,7 @@ void update_systemd_services_charts(
3274
// update the values
3275
struct cgroup *cg;
3276
for(cg = cgroup_root; cg ; cg = cg->next) {
3126
- if(unlikely(!cg->enabled || cg->pending_renames || !(cg->options & CGROUP_OPTIONS_SYSTEM_SLICE_SERVICE)))
3277
+ if(unlikely(!cg->enabled || cg->pending_renames || !is_cgroup_systemd_service(cg)))
3278
continue;
3279
3280
if(likely(do_cpu && cg->cpuacct_stat.updated)) {
@@ -3525,7 +3676,7 @@ void update_cgroup_charts(int update_every) {
3676
if(unlikely(!cg->enabled || cg->pending_renames))
3677
continue;
3678
3528
- if(likely(cgroup_enable_systemd_services && cg->options & CGROUP_OPTIONS_SYSTEM_SLICE_SERVICE)) {
3679
+ if(likely(cgroup_enable_systemd_services && is_cgroup_systemd_service(cg))) {
3680
if(cg->cpuacct_stat.updated && cg->cpuacct_stat.enabled == CONFIG_BOOLEAN_YES) services_do_cpu++;
3681
3682
if(cgroup_enable_systemd_services_detailed_memory && cg->memory.updated_detailed && cg->memory.enabled_detailed) services_do_mem_detailed++;
@@ -4542,6 +4693,7 @@ void *cgroups_main(void *ptr) {
4693
struct rusage thread;
4694
4695
if (getenv("KUBERNETES_SERVICE_HOST") != NULL && getenv("KUBERNETES_SERVICE_PORT") != NULL) {
4696
+ is_inside_k8s = 1;
4697
cgroup_enable_cpuacct_cpu_shares = CONFIG_BOOLEAN_YES;
4698
}
4699
@@ -4588,7 +4740,7 @@ void *cgroups_main(void *ptr) {
4740
if(unlikely(netdata_exit)) break;
4741
4742
find_dt += hb_dt;
4591
- if(unlikely(find_dt >= find_every || cgroups_check)) {
4743
+ if (unlikely(find_dt >= find_every || (!is_inside_k8s && cgroups_check))) {
4744
uv_cond_signal(&discovery_thread.cond_var);
4745
discovery_thread.start_discovery = 1;
4746
find_dt = 0;
@@ -4596,7 +4748,7 @@ void *cgroups_main(void *ptr) {
4748
}
4749
4750
uv_mutex_lock(&cgroup_root_mutex);
4599
- read_all_cgroups(cgroup_root);
4751
+ read_all_discovered_cgroups(cgroup_root);
4752
update_cgroup_charts(cgroup_update_every);
4753
uv_mutex_unlock(&cgroup_root_mutex);
4754
collectors/cgroups.plugin/sys_fs_cgroup.h
+1
-1
@@ -39,6 +39,6 @@ typedef struct netdata_ebpf_cgroup_shm {
39
40
#include "../proc.plugin/plugin_proc.h"
41
42
-extern char *parse_k8s_data(struct label **labels, char *data);
42
+extern char *k8s_parse_resolved_name(struct label **labels, char *data);
43
44
#endif //NETDATA_SYS_FS_CGROUP_H
collectors/cgroups.plugin/tests/test_cgroups_plugin.c
+4
-4
@@ -8,7 +8,7 @@ int netdata_zero_metrics_enabled = 1;
8
struct config netdata_config;
9
char *netdata_configured_primary_plugins_dir = NULL;
10
11
-static void test_parse_k8s_data(void **state)
11
+static void test_k8s_parse_resolved_name(void **state)
12
{
13
UNUSED(state);
14
@@ -89,7 +89,7 @@ static void test_parse_k8s_data(void **state)
89
expect_value(__wrap_add_label_to_list, label_source, LABEL_SOURCE_KUBERNETES);
90
}
91
92
- char *name = parse_k8s_data(&labels, data);
92
+ char *name = k8s_parse_resolved_name(&labels, data);
93
94
assert_string_equal(name, test_data[i].name);
95
assert_ptr_equal(labels, 0xff);
@@ -101,10 +101,10 @@ static void test_parse_k8s_data(void **state)
101
int main(void)
102
{
103
const struct CMUnitTest tests[] = {
104
- cmocka_unit_test(test_parse_k8s_data),
104
+ cmocka_unit_test(test_k8s_parse_resolved_name),
105
};
106
107
- int test_res = cmocka_run_group_tests_name("test_parse_k8s_data", tests, NULL, NULL);
107
+ int test_res = cmocka_run_group_tests_name("test_k8s_parse_resolved_name", tests, NULL, NULL);
108
109
return test_res;
110
}