Change default OOM score and scheduling policy to behave more sanely. (#12271)
Austin S. Hemmelgarn committed
Mar 11, 2022 at 11:22 UTC
8c8350371300d8fcafd794f1697bf22b83214120
4 files changed
+19
-74
daemon/README.md
+8
-51
@@ -254,57 +254,14 @@ where:
254
255
See [debugging](#debugging).
256
257
-## OOM Score
258
-
259
-Netdata runs with `OOMScore = 1000`. This means Netdata will be the first to be killed when your server runs out of
260
-memory.
261
-
262
-You can set Netdata OOMScore in `netdata.conf`, like this:
263
-
264
-```conf
265
-[global]
266
- OOM score = 1000
267
-```
268
-
269
-Netdata logs its OOM score when it starts:
270
-
271
-```sh
272
-# grep OOM /var/log/netdata/error.log
273
-2017-10-15 03:47:31: netdata INFO : Adjusted my Out-Of-Memory (OOM) score from 0 to 1000.
274
-```
275
-
276
-### OOM score and systemd
277
-
278
-Netdata will not be able to lower its OOM Score below zero, when it is started as the `netdata` user (systemd case).
279
-
280
-To allow Netdata control its OOM Score in such cases, you will need to edit `netdata.service` and set:
281
-
282
-```sh
283
-[Service]
284
-# The minimum Netdata Out-Of-Memory (OOM) score.
285
-# Netdata (via [global].OOM score in netdata.conf) can only increase the value set here.
286
-# To decrease it, set the minimum here and set the same or a higher value in netdata.conf.
287
-# Valid values: -1000 (never kill netdata) to 1000 (always kill netdata).
288
-OOMScoreAdjust=-1000
289
-```
290
-
291
-Run `systemctl daemon-reload` to reload these changes.
292
-
293
-The above, sets and OOMScore for Netdata to `-1000`, so that Netdata can increase it via `netdata.conf`.
294
-
295
-If you want to control it entirely via systemd, you can set in `netdata.conf`:
296
-
297
-```conf
298
-[global]
299
- OOM score = keep
300
-```
301
-
302
-Using the above, whatever OOM Score you have set at `netdata.service` will be maintained by netdata.
303
-
257
## Netdata process scheduling policy
258
306
-By default Netdata runs with the `idle` process scheduling policy, so that it uses CPU resources, only when there is
307
-idle CPU to spare. On very busy servers (or weak servers), this can lead to gaps on the charts.
259
+By default Netdata versions prior to 1.34.0 run with the `idle` process scheduling policy, so that it uses CPU
260
+resources, only when there is idle CPU to spare. On very busy servers (or weak servers), this can lead to gaps on
261
+the charts.
262
+
263
+Starting with version 1.34.0, Netdata instead uses the `batch` scheduling policy by default. This largely eliminates
264
+issues with gaps in charts on busy systems while still keeping the impact on the rest of the system low.
265
266
You can set Netdata scheduling policy in `netdata.conf`, like this:
267
@@ -315,9 +272,9 @@ You can set Netdata scheduling policy in `netdata.conf`, like this:
272
273
You can use the following:
274
318
-| policy | description |
275
+| policy | description |
276
| :-----------------------: | :---------- |
320
-| `idle` | use CPU only when there is spare - this is lower than nice 19 - it is the default for Netdata and it is so low that Netdata will run in "slow motion" under extreme system load, resulting in short (1-2 seconds) gaps at the charts. |
277
+| `idle` | use CPU only when there is spare - this is lower than nice 19 - it is the default for Netdata and it is so low that Netdata will run in "slow motion" under extreme system load, resulting in short (1-2 seconds) gaps at the charts. |
278
| `other`<br/>or<br/>`nice` | this is the default policy for all processes under Linux. It provides dynamic priorities based on the `nice` level of each process. Check below for setting this `nice` level for netdata. |
279
| `batch` | This policy is similar to `other` in that it schedules the thread according to its dynamic priority (based on the `nice` value). The difference is that this policy will cause the scheduler to always assume that the thread is CPU-intensive. Consequently, the scheduler will apply a small scheduling penalty with respect to wake-up behavior, so that this thread is mildly disfavored in scheduling decisions. |
280
| `fifo` | `fifo` can be used only with static priorities higher than 0, which means that when a `fifo` threads becomes runnable, it will always immediately preempt any currently running `other`, `batch`, or `idle` thread. `fifo` is a simple scheduling algorithm without time slicing. |
daemon/daemon.c
+7
-7
@@ -177,7 +177,7 @@ int become_user(const char *username, int pid_fd) {
177
178
static void oom_score_adj(void) {
179
char buf[30 + 1];
180
- long long int old_score, wanted_score = OOM_SCORE_ADJ_MAX, final_score = 0;
180
+ long long int old_score, wanted_score = 0, final_score = 0;
181
182
// read the existing score
183
if(read_single_signed_number_file("/proc/self/oom_score_adj", &old_score)) {
@@ -275,8 +275,8 @@ struct sched_def {
275
// the available members are important too!
276
// these are all the possible scheduling policies supported by netdata
277
278
-#ifdef SCHED_IDLE
279
- { "idle", SCHED_IDLE, 0, SCHED_FLAG_NONE },
278
+#ifdef SCHED_BATCH
279
+ { "batch", SCHED_BATCH, 0, SCHED_FLAG_USE_NICE },
280
#endif
281
282
#ifdef SCHED_OTHER
@@ -284,6 +284,10 @@ struct sched_def {
284
{ "nice", SCHED_OTHER, 0, SCHED_FLAG_USE_NICE },
285
#endif
286
287
+#ifdef SCHED_IDLE
288
+ { "idle", SCHED_IDLE, 0, SCHED_FLAG_NONE },
289
+#endif
290
+
291
#ifdef SCHED_RR
292
{ "rr", SCHED_RR, 0, SCHED_FLAG_PRIORITY_CONFIGURABLE },
293
#endif
@@ -292,10 +296,6 @@ struct sched_def {
296
{ "fifo", SCHED_FIFO, 0, SCHED_FLAG_PRIORITY_CONFIGURABLE },
297
#endif
298
295
-#ifdef SCHED_BATCH
296
- { "batch", SCHED_BATCH, 0, SCHED_FLAG_USE_NICE },
297
-#endif
298
-
299
// do not change the scheduling priority
300
{ "keep", 0, 0, SCHED_FLAG_KEEP_AS_IS },
301
{ "none", 0, 0, SCHED_FLAG_KEEP_AS_IS },
system/netdata.service.in
+2
-8
@@ -26,22 +26,16 @@ TimeoutStopSec=150
26
Restart=on-failure
27
RestartSec=30
28
29
-# The minimum netdata Out-Of-Memory (OOM) score.
30
-# netdata (via [global].OOM score in netdata.conf) can only increase the value set here.
31
-# To decrease it, set the minimum here and set the same or a higher value in netdata.conf.
32
-# Valid values: -1000 (never kill netdata) to 1000 (always kill netdata).
33
-OOMScoreAdjust=1000
34
-
29
# Valid policies: other (the system default) | batch | idle | fifo | rr
30
# To give netdata the max priority, set CPUSchedulingPolicy=rr and CPUSchedulingPriority=99
37
-CPUSchedulingPolicy=idle
31
+CPUSchedulingPolicy=batch
32
33
# This sets the scheduling priority (for policies: rr and fifo).
34
# Priority gets values 1 (lowest) to 99 (highest).
35
#CPUSchedulingPriority=1
36
37
# For scheduling policy 'other' and 'batch', this sets the lowest niceness of netdata (-20 highest to 19 lowest).
44
-#Nice=0
38
+Nice=0
39
40
# Capabilities
41
# is required for freeipmi and slabinfo plugins
system/netdata.service.v235.in
+2
-8
@@ -27,22 +27,16 @@ TimeoutStopSec=150
27
Restart=on-failure
28
RestartSec=30
29
30
-# The minimum netdata Out-Of-Memory (OOM) score.
31
-# netdata (via [global].OOM score in netdata.conf) can only increase the value set here.
32
-# To decrease it, set the minimum here and set the same or a higher value in netdata.conf.
33
-# Valid values: -1000 (never kill netdata) to 1000 (always kill netdata).
34
-OOMScoreAdjust=1000
35
-
30
# Valid policies: other (the system default) | batch | idle | fifo | rr
31
# To give netdata the max priority, set CPUSchedulingPolicy=rr and CPUSchedulingPriority=99
38
-CPUSchedulingPolicy=idle
32
+CPUSchedulingPolicy=batch
33
34
# This sets the scheduling priority (for policies: rr and fifo).
35
# Priority gets values 1 (lowest) to 99 (highest).
36
#CPUSchedulingPriority=1
37
38
# For scheduling policy 'other' and 'batch', this sets the lowest niceness of netdata (-20 highest to 19 lowest).
45
-#Nice=0
39
+Nice=0
40
41
[Install]
42
WantedBy=multi-user.target