@cryptotaxi247 / netdata-1 / commits / 9332a9f7d

Daemon docs edits (#19091)

Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud>

Fotis Voutsas committed Nov 27, 2024 at 21:39 UTC 9332a9f7d5f3e1c226855379a66e8380af301da0

3 files changed +176 -454

docs/DICTIONARY.md

@@ -10,6 +10,8 @@ When the context is clear, we can omit the "Netdata" prefix for brevity.

10 |------------------------|--------------------------------------------------------------------------|

11 | **Agent** (**Agents**) | The core monitoring software that collects, processes and stores metrics |

12 | **Cloud** | The centralized platform for managing and visualizing Netdata metrics |

13 +| **Daemon** | The main Netdata process |

14 +| **Collector(s)** | The various collectors of Netdata |

15 | **Registry** | The default Netdata Registry, or any Agent acting as one |

17 ## Database

src/daemon/README.md

+106 -348

@@ -1,431 +1,189 @@

1 -# Netdata daemon

1 +# Daemon

3 -The Netdata daemon is practically a synonym for the Netdata Agent, as it controls its

4 -entire operation. We support various methods to

5 -[start, stop, or restart the daemon](/docs/netdata-agent/start-stop-restart.md).

3 +The Netdata Daemon, often referred to as the Netdata Agent, controls the entire operation of the monitoring system. This document provides an overview of command-line options, debugging, and troubleshooting.

7 -This document provides some basic information on the command line options, log files, and how to debug and troubleshoot

5 +## Command-Line Options

9 -## Command line options

10 -

11 -Normally you don't need to supply any command line arguments to netdata.

12 -

13 -If you do though, they override the configuration equivalent options.

14 -

15 -To get a list of all command line parameters supported, run:

7 +While Netdata typically runs with default settings, you can override configurations using command-line options. For a complete list of options and detailed descriptions, run:

9 ```sh

10 netdata -h

11 ```

21 -The program will print the supported command line parameters.

22 -

23 -The command line options of the Netdata 1.10.0 version are the following:

24 -

25 -```sh

26 - ^

27 - |.-. .-. .-. .-. . netdata

28 - | '-' '-' '-' '-' real-time performance monitoring, done right!

29 - +----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+--->

30 -

32 - Released under GNU General Public License v3 or later.

34 -

35 - Home Page : https://netdata.cloud

36 - Source Code: https://github.com/netdata/netdata

37 - Docs : https://learn.netdata.cloud

38 - Support : https://github.com/netdata/netdata/issues

39 - License : https://github.com/netdata/netdata/blob/master/LICENSE.md

40 -

41 - Twitter : https://twitter.com/netdatahq

42 - LinkedIn : https://linkedin.com/company/netdata-cloud/

43 - Facebook : https://facebook.com/linuxnetdata/

44 -

45 -

46 - SYNOPSIS: netdata [options]

47 -

48 - Options:

49 -

50 - -c filename Configuration file to load.

51 - Default: /etc/netdata/netdata.conf

52 -

53 - -D Do not fork. Run in the foreground.

54 - Default: run in the background

13 +**Common Options**:

56 - -h Display this help message.

15 +| Option | Description | Default |

16 +|---------------|-----------------------------------|-----------------------------|

17 +| `-c filename` | Specify configuration file | `/etc/netdata/netdata.conf` |

18 +| `-D` | Run in foreground (do not fork) | Run in background |

19 +| `-d` | Run in background (fork) | Run in background |

20 +| `-P filename` | Save PID to file | No PID file |

21 +| `-i IP` | Set listening IP address | All IPv4 and IPv6 addresses |

22 +| `-p port` | Set API/Web port | `19999` |

23 +| `-s path` | Set prefix for `/proc` and `/sys` | No prefix |

24 +| `-t seconds` | Set internal clock interval | `1` |

25 +| `-u username` | Set running user | `netdata` |

26 +| `-v`, `-V` | Display version and exit | - |

27 +| `-W options` | Advanced options (see below) | - |

58 - -P filename File to save a pid while running.

59 - Default: do not save pid to a file

29 +## Logging

61 - -i IP The IP address to listen to.

62 - Default: all IP addresses IPv4 and IPv6

31 +For details about Netdata's logging system and configuration, see [Netdata Logging](/src/libnetdata/log/README.md).

64 - -p port API/Web port to use.

65 - Default: 19999

33 +## Process Scheduling Policy (Unix only)

67 - -s path Prefix for /proc and /sys (for containers).

68 - Default: no prefix

35 +Netdata uses the `batch` scheduling policy by default, which helps eliminate gaps in charts on busy systems while maintaining low system impact.

70 - -t seconds The internal clock of netdata.

71 - Default: 1

73 - -u username Run as user.

74 - Default: netdata

38 +<details>

39 +<summary>Change (Systemd)</summary>

76 - -v Print netdata version and exit.

41 +When Netdata runs under systemd as the `netdata` user, it can’t directly modify its scheduling policy and priority. Instead, configure these settings through systemd.

78 - -V Print netdata version and exit.

79 -

80 - -W options See Advanced options below.

81 -

82 -

83 - Advanced options:

84 -

85 - -W stacksize=N Set the stacksize (in bytes).

86 -

87 - -W debug_flags=N Set runtime tracing to debug.log.

88 -

89 - -W unittest Run internal unittests and exit.

90 -

91 - -W createdataset=N Create a DB engine dataset of N seconds and exit.

92 -

93 - -W set section option value

94 - set netdata.conf option from the command line.

95 -

96 - -W buildinfo Print the version, the configure options,

97 - a list of optional features, and whether they

98 - are enabled or not.

99 -

100 - -W buildinfojson Print the version, the configure options,

101 - a list of optional features, and whether they

102 - are enabled or not, in JSON format.

103 -

104 - -W simple-pattern pattern string

105 - Check if string matches pattern and exit.

106 -

107 - Signals netdata handles:

108 -

109 - - HUP Close and reopen log files.

110 - - USR2 Reload health configuration.

111 -```

43 +1. Use the following command to edit the systemd service (requires root privileges):

113 -You can send commands during runtime via [netdatacli](/src/cli/README.md).

45 + ```bash

46 + systemctl edit netdata

47 + ```

115 -## Log files

49 +2. Below are the available scheduling options. Uncomment and adjust the values according to your needs:

117 -Netdata uses 4 log files:

51 + ```bash

52 + [Service]

53 + ## CPU Scheduling Policy

54 + ## Options: other (system default) | batch | idle | fifo | rr

55 + #CPUSchedulingPolicy=other

119 -1. `error.log`

120 -2. `collector.log`

121 -3. `access.log`

122 -4. `debug.log`

57 + ## CPU Scheduling Priority (for fifo and rr policies)

58 + ## Range: 1 (lowest) to 99 (highest)

59 + ## Note: Netdata can only reduce this value via netdata.conf

60 + #CPUSchedulingPriority=1

124 -Any of them can be disabled by setting it to `/dev/null` or `none` in `netdata.conf`. By default `error.log`,

125 -`collector.log`, and `access.log` are enabled. `debug.log` is only enabled if debugging/tracing is also enabled

126 -(Netdata needs to be compiled with debugging enabled).

62 + ## Process Nice Level (for other and batch policies)

63 + ## Range: -20 (highest) to 19 (lowest)

64 + ## Note: Netdata can only increase this value via netdata.conf

65 + #Nice=0

66 + ```

128 -Log files are stored in `/var/log/netdata/` by default.

68 +3. Configure Netdata to preserve systemd settings by editing `netdata.conf`:

69 + ```text

70 + [global]

71 + process scheduling policy = keep

72 + ```

130 -### error.log

74 +4. [Restart](/docs/netdata-agent/start-stop-restart.md) netdata service.

132 -The `error.log` is the `stderr` of the `netdata` daemon .

76 +</details>

134 -For most Netdata programs (including standard external plugins shipped by netdata), the following lines may appear:

136 -| tag | description |

137 -|:-------:|:--------------------------------------------------------------------------------------------------------------------------|

138 -| `INFO` | Something important the user should know. |

139 -| `ERROR` | Something that might disable a part of netdata. The log line includes `errno` (if it is not zero). |

140 -| `FATAL` | Something prevented a program from running. The log line includes `errno` (if it is not zero) and the program exited. |

79 +<details>

80 +<summary>Change (Non-Systemd)</summary>

142 -The `FATAL` and `ERROR` messages will always appear in the logs, and `INFO`can be filtered using [severity level](/src/daemon/config/README.md#logs-section-options) option.

143 -

144 -So, when auto-detection of data collection fail, `ERROR` lines are logged and the relevant modules are disabled, but the

145 -program continues to run.

146 -

147 -When a Netdata program cannot run at all, a `FATAL` line is logged.

148 -

149 -### collector.log

150 -

151 -The `collector.log` is the `stderr` of all [collectors](/src/collectors/COLLECTORS.md)

152 - run by `netdata`.

153 -

154 -So if any process, in the Netdata process tree, writes anything to its standard error,

155 -it will appear in `collector.log`.

156 -

157 -Data stored inside this file follows pattern already described for `error.log`.

158 -

159 -### access.log

160 -

161 -The `access.log` logs web requests. The format is:

162 -

163 -```text

164 -DATE: ID: (sent/all = SENT_BYTES/ALL_BYTES bytes PERCENT_COMPRESSION%, prep/sent/total PREP_TIME/SENT_TIME/TOTAL_TIME ms): ACTION CODE URL

165 -```

166 -

167 -where:

168 -

169 -- `ID` is the client ID. Client IDs are auto-incremented every time a client connects to netdata.

170 -- `SENT_BYTES` is the number of bytes sent to the client, without the HTTP response header.

171 -- `ALL_BYTES` is the number of bytes of the response, before compression.

172 -- `PERCENT_COMPRESSION` is the percentage of traffic saved due to compression.

173 -- `PREP_TIME` is the time in milliseconds needed to prepared the response.

174 -- `SENT_TIME` is the time in milliseconds needed to sent the response to the client.

175 -- `TOTAL_TIME` is the total time the request was inside Netdata (from the first byte of the request to the last byte

176 - of the response).

177 -- `ACTION` can be `filecopy`, `options` (used in CORS), `data` (API call).

178 -

179 -### debug.log

180 -

181 -See [debugging](#debugging).

182 -

183 -## Netdata process scheduling policy

184 -

185 -By default Netdata versions prior to 1.34.0 run with the `idle` process scheduling policy, so that it uses CPU

186 -resources, only when there is idle CPU to spare. On very busy servers (or weak servers), this can lead to gaps on

187 -the charts.

188 -

189 -Starting with version 1.34.0, Netdata instead uses the `batch` scheduling policy by default. This largely eliminates

190 -issues with gaps in charts on busy systems while still keeping the impact on the rest of the system low.

191 -

192 -You can set Netdata scheduling policy in `netdata.conf`, like this:

82 +To modify the scheduling policy, [edit](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) `netdata.conf`:

84 ```text

85 [global]

86 process scheduling policy = idle

87 ```

199 -You can use the following:

89 +**Available Policies**:

201 -| policy | description |

202 -|:-------------------------:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

203 -| `idle` | use CPU only when there is spare - this is lower than nice 19 - it is the default for Netdata and it is so low that Netdata will run in "slow motion" under extreme system load, resulting in short (1-2 seconds) gaps at the charts. |

204 -| `other` or `nice` | this is the default policy for all processes under Linux. It provides dynamic priorities based on the `nice` level of each process. Check below for setting this `nice` level for netdata. |

205 -| `batch` | This policy is similar to `other` in that it schedules the thread according to its dynamic priority (based on the `nice` value). The difference is that this policy will cause the scheduler to always assume that the thread is CPU-intensive. Consequently, the scheduler will apply a small scheduling penalty with respect to wake-up behavior, so that this thread is mildly disfavored in scheduling decisions. |

206 -| `fifo` | `fifo` can be used only with static priorities higher than 0, which means that when a `fifo` threads becomes runnable, it will always immediately preempt any currently running `other`, `batch`, or `idle` thread. `fifo` is a simple scheduling algorithm without time slicing. |

207 -| `rr` | a simple enhancement of `fifo`. Everything described above for `fifo` also applies to `rr`, except that each thread is allowed to run only for a maximum time quantum. |

208 -| `keep` or `none` | do not set scheduling policy, priority or nice level - i.e. keep running with whatever it is set already (e.g. by systemd). |

91 +| Policy | Description |

92 +|----------------|---------------------------------------------------------------------------------------------------------------------------|

93 +| `batch` | Similar to `other` but treats the thread as CPU-intensive, applying a mild scheduling penalty. This is Netdata's default. |

94 +| `idle` | Uses CPU only when available (lower than nice 19). Under extreme system load, may cause 1-2 second gaps in charts. |

95 +| `other`/`nice` | Linux's default process policy. Uses dynamic priorities based on the process's `nice` level. |

96 +| `fifo` | Requires static priorities above 0. Immediately preempts `other`, `batch`, or `idle` threads. No time slicing. |

97 +| `rr` | Enhanced `fifo` with maximum time quantum for each thread. |

98 +| `keep`/`none` | Maintains existing scheduling policy and priority settings. |

210 -For more information see `man sched`.

100 +For additional details about process scheduling, see [man sched](https://man7.org/linux/man-pages/man7/sched.7.html).

101

212 -### Scheduling priority for `rr` and `fifo`

102 +**FIFO and RR Priority**:

103

214 -Once the policy is set to one of `rr` or `fifo`, the following will appear:

104 +When using `fifo` or `rr` policies, you can set the process priority in `netdata.conf`:

105

106 ```text

107 [global]

108 process scheduling priority = 0

109 ```

110

221 -These priorities are usually from 0 to 99. Higher numbers make the process more

222 -important.

111 +Priority values range from 0 to 99, with higher values indicating higher process importance.

112

224 -### nice level for policies `other` or `batch`

113 +**Nice Level**

114

226 -When the policy is set to `other`, `nice`, or `batch`, the following will appear:

115 +For `other`, `nice`, or `batch` policies, you can adjust the nice level:

116

117 ```text

118 [global]

119 process nice level = 19

120 ```

121

233 -## Scheduling settings and systemd

234 -

235 -Netdata will not be able to set its scheduling policy and priority to more important values when it is started as the

236 -`netdata` user (systemd case).

237 -

238 -You can set these settings at `/etc/systemd/system/netdata.service`:

239 -

240 -```sh

241 -[Service]

242 -# By default Netdata switches to scheduling policy idle, which makes it use CPU, only

243 -# when there is spare available.

244 -# Valid policies: other (the system default) | batch | idle | fifo | rr

245 -#CPUSchedulingPolicy=other

246 -

247 -# This sets the maximum scheduling priority Netdata can set (for policies: rr and fifo).

248 -# Netdata (via [global].process scheduling priority in netdata.conf) can only lower this value.

249 -# Priority gets values 1 (lowest) to 99 (highest).

250 -#CPUSchedulingPriority=1

251 -

252 -# For scheduling policy 'other' and 'batch', this sets the lowest niceness of netdata.

253 -# Netdata (via [global].process nice level in netdata.conf) can only increase the value set here.

254 -#Nice=0

255 -```

256 -

257 -Run `systemctl daemon-reload` to reload these changes.

258 -

259 -Now, tell Netdata to keep these settings, as set by systemd, by editing

260 -`netdata.conf` and setting:

261 -

262 -```text

263 -[global]

264 - process scheduling policy = keep

265 -```

266 -

267 -Using the above, whatever scheduling settings you have set at `netdata.service`

268 -will be maintained by netdata.

269 -

270 -### Example 1: Netdata with nice -1 on non-systemd systems

271 -

272 -On a system that is not based on systemd, to make Netdata run with nice level -1 (a little bit higher to the default for

273 -all programs), edit `netdata.conf` and set:

274 -

275 -```text

276 -[global]

277 - process scheduling policy = other

278 - process nice level = -1

279 -```

280 -

281 -then [restart Netdata](/docs/netdata-agent/start-stop-restart.md):

282 -

283 -#### Example 2: Netdata with nice -1 on systemd systems

284 -

285 -On a system that is based on systemd, to make Netdata run with nice level -1 (a little bit higher to the default for all

286 -programs), edit `netdata.conf` and set:

287 -

288 -```text

289 -[global]

290 - process scheduling policy = keep

291 -```

292 -

293 -edit /etc/systemd/system/netdata.service and set:

294 -

295 -```sh

296 -[Service]

297 -CPUSchedulingPolicy=other

298 -Nice=-1

299 -```

300 -

301 -then execute:

302 -

303 -```sh

304 -sudo systemctl daemon-reload

305 -sudo systemctl restart netdata

306 -```

307 -

308 -## Virtual memory

122 +The nice level ranges from -20 (the highest priority) to 19 (the lowest priority). A higher value means the process is "nicer" to other processes by using fewer CPU resources.

123

310 -You may notice that netdata's virtual memory size, as reported by `ps` or `/proc/pid/status` (or even netdata's

311 -applications virtual memory chart) is unrealistically high.

312 -

313 -For example, it may be reported to be 150+MB, even if the resident memory size is just 25MB. Similar values may be

314 -reported for Netdata plugins too.

315 -

316 -Check this for example: A Netdata installation with default settings on Ubuntu

317 -16.04LTS. The top chart is **real memory used**, while the bottom one is

318 -**virtual memory**:

319 -

320 -![image](https://cloud.githubusercontent.com/assets/2662304/19013772/5eb7173e-87e3-11e6-8f2b-a2ccfeb06faf.png)

321 -

322 -### Why does this happen?

323 -

324 -The system memory allocator allocates virtual memory arenas, per thread running. On Linux systems this defaults to 16MB

325 -per thread on 64 bit machines. So, if you get the difference between real and virtual memory and divide it by 16MB you

326 -will roughly get the number of threads running.

327 -

328 -The system does this for speed. Having a separate memory arena for each thread, allows the threads to run in parallel in

329 -multi-core systems, without any locks between them.

330 -

331 -This behavior is system specific. For example, the chart above when running

332 -Netdata on Alpine Linux (that uses **musl** instead of **glibc**) is this:

333 -

334 -![image](https://cloud.githubusercontent.com/assets/2662304/19013807/7cf5878e-87e4-11e6-9651-082e68701eab.png)

335 -

336 -### Can we do anything to lower it?

337 -

338 -Since Netdata already uses minimal memory allocations while it runs (i.e. it adapts its memory on start, so that while

339 -repeatedly collects data it does not do memory allocations), it already instructs the system memory allocator to

340 -minimize the memory arenas for each thread. We have also added [2 configuration

341 -options](https://github.com/netdata/netdata/blob/5645b1ee35248d94e6931b64a8688f7f0d865ec6/src/main.c#L410-L418) to allow

342 -you tweak these settings: `glibc malloc arena max for plugins` and `glibc malloc arena max for netdata`.

343 -

344 -However, even if we instructed the memory allocator to use just one arena, it

345 -seems it allocates an arena per thread.

346 -

347 -Netdata also supports `jemalloc` and `tcmalloc`, however both behave exactly the

348 -same to the glibc memory allocator in this aspect.

349 -

350 -### Is this a problem?

351 -

352 -No, it is not.

353 -

354 -Linux reserves real memory (physical RAM) in pages (on x86 machines pages are 4KB each). So even if the system memory

355 -allocator is allocating huge amounts of virtual memory, only the 4KB pages that are actually used are reserving physical

356 -RAM. The **real memory** chart on Netdata application section, shows the amount of physical memory these pages occupy(it

357 -accounts the whole pages, even if parts of them are actually used).

124 +</details>

125

126 ## Debugging

127

361 -When you compile Netdata with debugging:

362 -

363 -1. compiler optimizations for your CPU are disabled (Netdata will run somewhat slower)

364 -

365 -2. a lot of code is added all over netdata, to log debug messages to `/var/log/netdata/debug.log`. However, nothing is

366 - printed by default. Netdata allows you to select which sections of Netdata you want to trace. Tracing is activated

367 - via the config option `debug flags`. It accepts a hex number, to enable or disable specific sections. You can find

368 - the options supported at [log.h](https://raw.githubusercontent.com/netdata/netdata/master/src/libnetdata/log/log.h).

369 - They are the `D_*` defines. The value `0xffffffffffffffff` will enable all possible debug flags.

128 +When Netdata is compiled with debugging enabled:

129

371 -Once Netdata is compiled with debugging and tracing is enabled for a few sections, the file `/var/log/netdata/debug.log`

372 -will contain the messages.

130 +- **Performance Impact**: Compiler optimizations are disabled, which may result in slightly reduced performance.

131 +- **Debug Logging**: Disabled by default. To enable logging for specific components:

132 + - Open `netdata.conf`.

133 + - Set the `debug flags` option to a hex value that corresponds to the components you want to trace.

134 + - Debug flag options are defined in [log.h](https://raw.githubusercontent.com/netdata/netdata/master/src/libnetdata/log/log.h) as `D_*` values. Use `0xffffffffffffffff` to enable all debug flags.

135

374 -> Do not forget to disable tracing (`debug flags = 0`) when you are done tracing. The file `debug.log` can grow too

375 -> fast.

136 +> **Important**

137 +>

138 +> Remember to disable debug logging (`debug flags = 0`) after you finish troubleshooting. Debug logs can grow rapidly and consume significant disk space.

139

140 ### Compiling Netdata with debugging

141

379 -To compile Netdata with debugging, use this:

142 +To compile Netdata with debugging capabilities:

143

144 ```sh

382 -# step into the Netdata source directory

145 +# Navigate to Netdata source directory

146 cd /usr/src/netdata.git

147

385 -# run the installer with debugging enabled

148 +# Install with debugging enabled

149 CFLAGS="-O1 -ggdb -DNETDATA_INTERNAL_CHECKS=1" ./netdata-installer.sh

150 ```

151

389 -The above will compile and install Netdata with debugging info embedded. You can now use `debug flags` to set the

390 -section(s) you need to trace.

391 -

392 -### Debugging crashes

393 -

394 -We have made the most to make Netdata crash free. If however, Netdata crashes on your system, it would be very helpful

395 -to provide stack traces of the crash. Without them, is will be almost impossible to find the issue (the code base is

396 -quite large to find such an issue by just observing it).

397 -

398 -To provide stack traces, **you need to have Netdata compiled with debugging**. There is no need to enable any tracing

399 -(`debug flags`).

152 +After installation, use the `debug flags` setting in your configuration to specify which components to trace.

153

401 -Then you need to be in one of the following 2 cases:

154 +This compilation method includes debugging information in the binary and enables internal checks. **This is recommended only for development or troubleshooting purposes**.

155

403 -1. Netdata crashes and you have a core dump

156 +### Debugging Crashes

157

405 -2. you can reproduce the crash

158 +While Netdata is designed to be highly stable, if you encounter a crash, providing stack traces greatly helps in identifying the root cause.

159

407 -If you are not on these cases, you need to find a way to be (i.e. if your system does not produce core dumps, check your

408 -distro documentation to enable them).

160 +To generate useful debugging information, ensure you have Netdata [compiled with debugging enabled](#debugging).

161

410 -### Netdata crashes and you have a core dump

162 +#### Method 1: Analyzing a Core Dump

163

412 -> you need to have Netdata compiled with debugging info for this to work (check above)

413 -

414 -Run the following command and post the output on a github issue.

164 +If you have a core dump from the crash, run:

165

166 ```sh

167 gdb $(which netdata) /path/to/core/dump

168 ```

169

420 -### You can reproduce a Netdata crash on your system

170 +#### Method 2: Using Valgrind for Reproducible Crashes

171

422 -> you need to have Netdata compiled with debugging info for this to work (check above)

172 +If you can reproduce the crash consistently:

173

424 -Install the package `valgrind` and run:

174 +1. Install the `valgrind` package

175 +2. Run Netdata under Valgrind:

176

426 -```sh

427 -valgrind $(which netdata) -D

428 -```

177 + ```sh

178 + valgrind $(which netdata) -D

179 + ```

180 +

181 +Netdata will run significantly slower under Valgrind. When the crash occurs, Valgrind will output the stack trace to your console.

182 +

183 +#### Reporting the Issue

184 +

185 +For either method:

186

430 -Netdata will start and it will be a lot slower. Now reproduce the crash and `valgrind` will dump on your console the

431 -stack trace. Open a new github issue and post the output.

187 +- Create a [new GitHub issue](https://github.com/netdata/netdata/issues/new/choose).

188 +- Include the complete output from gdb or Valgrind.

189 +- Add any relevant details about the circumstances of the crash.

src/daemon/config/README.md

+68 -106

@@ -1,90 +1,63 @@

1 -# Daemon configuration

2 -

3 -<details>

4 -<summary>The daemon configuration file is read from /etc/netdata/netdata.conf.</summary>

5 -

6 -Depending on your installation method, Netdata will have been installed either directly under `/`, or

7 -under `/opt/netdata`. The paths mentioned here and in the documentation in general assume that your installation is

8 -under `/`. If it is not, you will find the exact same paths under `/opt/netdata` as well. (i.e. `/etc/netdata` will

9 -be `/opt/netdata/etc/netdata`).

10 -

11 -</details>

12 -

13 -This config file **is not needed by default**. Netdata works fine out of the box without it. But it does allow you to

14 -adapt the general behavior of Netdata, in great detail. You can find all these settings, with their default values, by

15 -accessing the URL `https://netdata.server.hostname:19999/netdata.conf`. For example check the configuration file

16 -of [netdata.firehol.org](http://netdata.firehol.org/netdata.conf). HTTP access to this file is limited by default to

17 -[private IPs](https://en.wikipedia.org/wiki/Private_network), via

18 -the [web server access lists](/src/web/server/README.md#access-lists).

19 -

20 -`netdata.conf` has sections stated with `[section]`. You will see the following sections:

21 -

22 -1. `[global]` to [configure](#global-section-options) the [Netdata daemon](/src/daemon/README.md).

23 -2. `[db]` to [configure](#db-section-options) the database of Netdata.

24 -3. `[directories]` to [configure](#directories-section-options) the directories used by Netdata.

25 -4. `[logs]` to [configure](#logs-section-options) the Netdata logging.

26 -5. `[environment variables]` to [configure](#environment-variables-section-options) the environment variables used

27 - Netdata.

28 -6. `[sqlite]` to [configure](#sqlite-section-options) the [Netdata daemon](/src/daemon/README.md) SQLite settings.

29 -7. `[ml]` to configure settings for [machine learning](/src/ml/README.md).

30 -8. `[health]` to [configure](#health-section-options) general settings for [health monitoring](/src/health/README.md).

31 -9. `[web]` to [configure the web server](/src/web/server/README.md).

32 -10. `[registry]` for the [Netdata Registry](/src/registry/README.md).

33 -11. `[global statistics]` for the [Netdata Registry](/src/registry/README.md).

34 -12. `[statsd]` for the general settings of the [stats.d.plugin](/src/collectors/statsd.plugin/README.md).

35 -13. `[plugins]` to [configure](#plugins-section-options) which [collectors](/src/collectors/README.md) to use and PATH

36 - settings.

37 -14. `[plugin:NAME]` sections for each collector plugin, under the

38 - comment [Per plugin configuration](#per-plugin-configuration).

39 -

40 -The configuration file is a `name = value` dictionary. Netdata will not complain if you set options unknown to it. When

41 -you check the running configuration by accessing the URL `/netdata.conf` on your Netdata server, Netdata will add a

42 -comment on settings it does not currently use.

1 +# Daemon Configuration Reference

2 +

3 +The Netdata daemon's main configuration file is located at `/INSTALL_PREFIX/netdata/netdata.conf`. While Netdata works effectively with default settings, this file allows you to fine-tune its behavior.

4 +

5 +You can view your current configuration, including default values, at `http://IP:19999/netdata.conf`. Access to this URL is [restricted to local IPs by default](/src/web/server/README.md#access-lists).

6 +

7 +The configuration file uses an INI-style format with `[SECTION]` headers:

8 +

9 +| Section | Controls |

10 +|-------------------------------------------------------------------|----------------------------------------------------------|

11 +| [[global]](#global-section-options) | [Daemon](/src/daemon/README.md) |

12 +| [[db]](#db-section-options) | [Database](/src/database/README.md) |

13 +| [[directories]](#directories-section-options) | Directories used by Netdata |

14 +| [[logs]](#logs-section-options) | Logging |

15 +| [[environment variables]](#environment-variables-section-options) | Environment variables |

16 +| [[sqlite]](#sqlite-section-options) | SQLite |

17 +| `[ml]` | [Machine Learning](/src/ml/README.md) |

18 +| [[health]](#health-section-options) | [Health monitoring](/src/health/README.md) |

19 +| `[web]` | [Web Server](/src/web/server/README.md) |

20 +| `[registry]` | [Registry](/src/registry/README.md) |

21 +| `[global statistics]` | Internal monitoring |

22 +| `[statsd]` | [StatsD plugin](/src/collectors/statsd.plugin/README.md) |

23 +| [`[plugins]`](#plugins-section-options) | Data collection Plugins (Collectors) |

24 +| [[plugin:NAME]](#per-plugin-configuration) | Individual [Plugins](#per-plugin-configuration) |

25 +

26 +> **Note**

27 +>

28 +> The configuration uses a simple `name = value` format. Netdata tolerates unknown options, marking them with comments when viewing the running configuration through `/netdata.conf`.

30 ## Applying changes

46 -After `netdata.conf` has been modified, Netdata needs to be [restarted](/docs/netdata-agent/start-stop-restart.md) for

47 -changes to apply:

48 -

49 -```bash

50 -sudo systemctl restart netdata

51 -```

52 -

53 -If the above does not work, try the following:

32 +After `netdata.conf` has been modified, Netdata needs to be [restarted](/docs/netdata-agent/start-stop-restart.md) for changes to apply.

55 -```bash

56 -sudo killall netdata; sleep 10; sudo netdata

57 -```

34 +## Configuration Sections

59 -Please note that your data history will be lost if you have modified `history` parameter in section `[global]`.

60 -

61 -## Sections

62 -

63 -### [global] section options

36 +### `global` section options

38 | setting | default | info |

39 |:----------------------------------:|:-------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

67 -| process scheduling policy | `keep` | See [Netdata process scheduling policy](/src/daemon/README.md#netdata-process-scheduling-policy) |

40 +| process scheduling policy | `keep` | See [Netdata process scheduling policy](/src/daemon/README.md#process-scheduling-policy-unix-only) |

41 | OOM score | `0` | |

69 -| glibc malloc arena max for plugins | `1` | See [Virtual memory](/src/daemon/README.md#virtual-memory). |

70 -| glibc malloc arena max for Netdata | `1` | See [Virtual memory](/src/daemon/README.md#virtual-memory). |

42 +| glibc malloc arena max for plugins | `1` | |

43 +| glibc malloc arena max for Netdata | `1` | |

44 | hostname | auto-detected | The hostname of the computer running Netdata. |

72 -| host access prefix | empty | This is used in docker environments where /proc, /sys, etc have to be accessed via another path. You may also have to set SYS_PTRACE capability on the docker for this work. Check [issue 43](https://github.com/netdata/netdata/issues/43). |

45 +| host access prefix | empty | This is used in Docker environments where /proc, /sys, etc have to be accessed via another path. You may also have to set SYS_PTRACE capability on the docker for this work. Check [issue 43](https://github.com/netdata/netdata/issues/43). |

46 | timezone | auto-detected | The timezone retrieved from the environment variable |

47 | run as user | `netdata` | The user Netdata will run as. |

48 | pthread stack size | auto-detected | |

77 -### [db] section options

50 +### `db` section options

52 | setting | default | info |

53 |:---------------------------------------------:|:------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

81 -| mode | `dbengine` | `dbengine`: The default for long-term metrics storage with efficient RAM and disk usage. Can be extended with `dbengine page cache size` and `dbengine tier X retention size`. `ram`: The round-robin database will be temporary and it will be lost when Netdata exits. `alloc`: Similar to `ram`, but can significantly reduce memory usage, when combined with a low retention and does not support KSM. `none`: Disables the database at this host, and disables health monitoring entirely, as that requires a database of metrics. Not to be used together with streaming. |

54 +| mode | `dbengine` | `dbengine`: The default for long-term metrics storage with efficient RAM and disk usage. Can be extended with `dbengine page cache size` and `dbengine tier X retention size`. `ram`: The round-robin database will be temporary and it will be lost when Netdata exits. `alloc`: Similar to `ram`, but can significantly reduce memory usage, when combined with a low retention and does not support KSM. `none`: Disables the database at this host, and disables Health monitoring entirely, as that requires a database of metrics. Not to be used together with streaming. |

55 | retention | `3600` | Used with `mode = ram/alloc`, not the default `mode = dbengine`. This number reflects the number of entries the `netdata` daemon will by default keep in memory for each chart dimension. Check [Memory Requirements](/docs/netdata-agent/sizing-netdata-agents/disk-requirements-and-retention.md) for more information. |

83 -| storage tiers | `3` | The number of storage tiers you want to have in your dbengine. Check the tiering mechanism in the [dbengine's reference](/src/database/engine/README.md#tiering). You can have up to 5 tiers of data (including the _Tier 0_). This number ranges between 1 and 5. |

56 +| storage tiers | `3` | The number of storage tiers you want to have in your dbengine. Check the tiering mechanism in the [dbengine's reference](/src/database/engine/README.md#tiers). You can have up to 5 tiers of data (including the _Tier 0_). This number ranges between 1 and 5. |

57 | dbengine page cache size | `32MiB` | Determines the amount of RAM in MiB that is dedicated to caching for _Tier 0_ Netdata metric values. |

58 | dbengine tier **`N`** retention size | `1GiB` | The disk space dedicated to metrics storage, per tier. Can be used in single-node environments as well. `N belongs to [1..4]` |

59 | dbengine tier **`N`** retention time | `14d`, `3mo`, `1y`, `1y`, `1y` | The database retention, expressed in time. Can be used in single-node environments as well. `N belongs to [1..4]` |

87 -| update every | `1` | The frequency in seconds, for data collection. For more information see the [performance guide](/docs/netdata-agent/configuration/optimize-the-netdata-agents-performance.md). These metrics stored as _Tier 0_ data. Explore the tiering mechanism in the [dbengine's reference](/src/database/engine/README.md#tiering). |

60 +| update every | `1` | The frequency in seconds, for data collection. For more information see the [performance guide](/docs/netdata-agent/configuration/optimize-the-netdata-agents-performance.md). These metrics stored as _Tier 0_ data. Explore the tiering mechanism in the [dbengine's reference](/src/database/engine/README.md#tiers). |

61 | dbengine tier **`N`** update every iterations | `60` | The down sampling value of each tier from the previous one. For each Tier, the greater by one Tier has N (equal to 60 by default) less data points of any metric it collects. This setting can take values from `2` up to `255`. `N belongs to [1..4]` |

62 | dbengine tier back fill | `new` | Specifies the strategy of recreating missing data on higher database Tiers. `new`: Sees the latest point on each Tier and save new points to it only if the exact lower Tier has available points for it's observation window (`dbengine tier N update every iterations` window). `none`: No back filling is applied. `N belongs to [1..4]` |

63 | memory deduplication (ksm) | `yes` | When set to `yes`, Netdata will offer its in-memory round robin database and the dbengine page cache to kernel same page merging (KSM) for deduplication. |

@@ -95,26 +68,26 @@ Please note that your data history will be lost if you have modified `history` p

69 > ### Info

70 >

98 ->The multiplication of all the **enabled** tiers `dbengine tier N update every iterations` values must be less than `65535`.

71 +> The multiplication of all the **enabled** tiers `dbengine tier N update every iterations` values must be less than `65535`.

100 -### [directories] section options

73 +### `directories` section options

75 | setting | default | info |

76 |:-------------------:|:------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

77 | config | `/etc/netdata` | The directory configuration files are kept. |

78 | stock config | `/usr/lib/netdata/conf.d` | |

106 -| log | `/var/log/netdata` | The directory in which the [log files](/src/daemon/README.md#log-files) are kept. |

79 +| log | `/var/log/netdata` | The directory in which the [log files](/src/daemon/README.md#logging) are kept. |

80 | web | `/usr/share/netdata/web` | The directory the web static files are kept. |

81 | cache | `/var/cache/netdata` | The directory the memory database will be stored if and when Netdata exits. Netdata will re-read the database when it will start again, to continue from the same point. |

109 -| lib | `/var/lib/netdata` | Contains the alert log and the Netdata instance GUID. |

82 +| lib | `/var/lib/netdata` | Contains the Alert log and the Netdata instance GUID. |

83 | home | `/var/cache/netdata` | Contains the db files for the collected metrics. |

84 | lock | `/var/lib/netdata/lock` | Contains the data collectors lock files. |

85 | plugins | `"/usr/libexec/netdata/plugins.d" "/etc/netdata/custom-plugins.d"` | The directory plugin programs are kept. This setting supports multiple directories, space separated. If any directory path contains spaces, enclose it in single or double quotes. |

113 -| health config | `/etc/netdata/health.d` | The directory containing the user alert configuration files, to override the stock configurations |

114 -| stock health config | `/usr/lib/netdata/conf.d/health.d` | Contains the stock alert configuration files for each collector |

86 +| Health config | `/etc/netdata/health.d` | The directory containing the user Alert configuration files, to override the stock configurations |

87 +| stock Health config | `/usr/lib/netdata/conf.d/health.d` | Contains the stock Alert configuration files for each collector |

88 | registry | `/opt/netdata/var/lib/netdata/registry` | Contains the [registry](/src/registry/README.md) database and GUID that uniquely identifies each Netdata Agent |

117 -### [logs] section options

90 +### `logs` section options

92 There are additional configuration options for the logs. For more info, see [Netdata Logging](/src/libnetdata/log/README.md).

@@ -125,14 +98,14 @@ There are additional configuration options for the logs. For more info, see [Net

98 | error | `/var/log/netdata/error.log` | The filename to save error messages for Netdata daemon and all plugins (`stderr` is sent here for all Netdata programs, including the plugins). You can also set it to `syslog` to send the errors to syslog, or `off` to disable this log. |

99 | access | `/var/log/netdata/access.log` | The filename to save the log of web clients accessing Netdata charts. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. |

100 | collector | `journal` | The filename to save the log of Netdata collectors. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |

128 -| health | `journal` | The filename to save the log of Netdata health collectors. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |

101 +| Health | `journal` | The filename to save the log of Netdata Health collectors. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |

102 | daemon | `journal` | The filename to save the log of Netdata daemon. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |

103 | facility | `daemon` | A facility keyword is used to specify the type of system that is logging the message. |

104 | logs flood protection period | `1m` | Length of period during which the number of errors should not exceed the `errors to trigger flood protection`. |

105 | logs to trigger flood protection | `1000` | Number of errors written to the log in `errors flood protection period` sec before flood protection is activated. |

106 | level | `info` | Controls which log messages are logged, with error being the most important. Supported values: `info` and `error`. |

107

135 -### [environment variables] section options

108 +### `environment variables` section options

109

110 | setting | default | info |

111 |:----------:|:-----------------:|:-----------------------------------------------------------|

@@ -140,7 +113,7 @@ There are additional configuration options for the logs. For more info, see [Net

113 | PATH | `auto-detected` | Specifies the directories to be searched to find a command |

114 | PYTHONPATH | | Used to set a custom python path |

115

143 -### [sqlite] section options

116 +### `sqlite` section options

117

118 | setting | default | info |

119 |:------------------:|:-------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

@@ -151,35 +124,32 @@ There are additional configuration options for the logs. For more info, see [Net

124 | journal size limit | `16777216` | Used to set a new [limit in bytes for the database](https://www.sqlite.org/pragma.html#pragma_journal_size_limit) |

125 | cache size | `-2000` | Used to [suggest the maximum number of database disk pages](https://www.sqlite.org/pragma.html#pragma_cache_size) that SQLite will hold in memory at once per open database file |

126

154 -### [health] section options

127 +### `health` section options

128

156 -This section controls the general behavior of the health monitoring capabilities of Netdata.

129 +This section controls the general behavior of the Health monitoring capabilities of Netdata.

130

158 -Specific alerts are configured in per-collector config files under the `health.d` directory. For more info, see [health

131 +Specific Alerts are configured in per-collector config files under the `health.d` directory. For more info, see [health

132 monitoring](/src/health/README.md).

133

134 [Alert notifications](/src/health/notifications/README.md) are configured in `health_alarm_notify.conf`.

135

136 | setting | default | info |

137 |:--------------------------------------:|:------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

165 -| enabled | `yes` | Set to `no` to disable all alerts and notifications |

166 -| in memory max health log entries | 1000 | Size of the alert history held in RAM |

167 -| script to execute on alarm | `/usr/libexec/netdata/plugins.d/alarm-notify.sh` | The script that sends alert notifications. Note that in versions before 1.16, the plugins.d directory may be installed in a different location in certain OSs (e.g. under `/usr/lib/netdata`). |

168 -| run at least every | `10s` | Controls how often all alert conditions should be evaluated. |

169 -| postpone alarms during hibernation for | `1m` | Prevents false alerts. May need to be increased if you get alerts during hibernation. |

170 -| health log retention | `5d` | Specifies the history of alert events (in seconds) kept in the Agent's sqlite database. |

171 -| enabled alarms | * | Defines which alerts to load from both user and stock directories. This is a [simple pattern](/src/libnetdata/simple_pattern/README.md) list of alert or template names. Can be used to disable specific alerts. For example, `enabled alarms = !oom_kill *` will load all alerts except `oom_kill`. |

138 +| enabled | `yes` | Set to `no` to disable all Alerts and notifications |

139 +| in memory max Health log entries | 1000 | Size of the Alert history held in RAM |

140 +| script to execute on alarm | `/usr/libexec/netdata/plugins.d/alarm-notify.sh` | The script that sends Alert notifications. Note that in versions before 1.16, the plugins.d directory may be installed in a different location in certain OSs (e.g. under `/usr/lib/netdata`). |

141 +| run at least every | `10s` | Controls how often all Alert conditions should be evaluated. |

142 +| postpone alarms during hibernation for | `1m` | Prevents false Alerts. May need to be increased if you get Alerts during hibernation. |

143 +| Health log retention | `5d` | Specifies the history of Alert events (in seconds) kept in the Agent's sqlite database. |

144 +| enabled alarms | * | Defines which Alerts to load from both user and stock directories. This is a [simple pattern](/src/libnetdata/simple_pattern/README.md) list of Alert or template names. Can be used to disable specific Alerts. For example, `enabled alarms = !oom_kill *` will load all Alerts except `oom_kill`. |

145

173 -### [web] section options

146 +### `web` section options

147

148 Refer to the [web server documentation](/src/web/server/README.md)

149

177 -### [plugins] section options

150 +### `plugins` section options

151

179 -In this section you will see be a boolean (`yes`/`no`) option for each plugin (e.g. tc, cgroups, apps, proc etc.). Note

180 -that the configuration options in this section for the orchestrator plugins `python.d` and `charts.d` control **all the

181 -modules** written for that orchestrator. For instance, setting `python.d = no` means that all Python modules

182 -under `collectors/python.d.plugin` will be disabled.

152 +In this section you will see be a boolean (`yes`/`no`) option for each plugin (e.g., tc, cgroups, apps, proc etc.). Note that the configuration options in this section for the orchestrator plugins `python.d` and `charts.d` control **all the modules** written for that orchestrator. For instance, setting `python.d = no` means that all Python modules under `collectors/python.d.plugin` will be disabled.

153

154 Additionally, there will be the following options:

155

@@ -189,10 +159,9 @@ Additionally, there will be the following options:

159 | check for new plugins every | 60 | The time in seconds to check for new plugins in the plugins directory. This allows having other applications dynamically creating plugins for Netdata. |

160 | checks | `no` | This is a debugging plugin for the internal latency |

161

192 -### [registry] section options

162 +### `registry` section options

163

194 -To understand what this section is and how it should be configured, please refer to

195 -the [registry documentation](/src/registry/README.md).

164 +To understand what this section is and how it should be configured, refer to the [registry documentation](/src/registry/README.md).

165

166 ## Per-plugin configuration

167

@@ -200,25 +169,18 @@ The configuration options for plugins appear in sections following the pattern `

169

170 ### Internal plugins

171

203 -Most internal plugins will provide additional options. Check [Internal Plugins](/src/collectors/README.md) for more

204 -information.

172 +Most internal plugins will provide additional options. Check [Internal Plugins](/src/collectors/README.md) for more information.

173

206 -Please note, that by default Netdata will enable monitoring metrics for disks, memory, and network only when they are

207 -not zero. If they are constantly zero they are ignored. Metrics that will start having values, after Netdata is started,

208 -will be detected and charts will be automatically added to the dashboard (a refresh of the dashboard is needed for them

209 -to appear though). Use `yes` instead of `auto` in plugin configuration sections to enable these charts permanently. You

210 -can also set the `enable zero metrics` option to `yes` in the `[global]` section which enables charts with zero metrics

174 +Note that by default, Netdata will enable monitoring metrics for disks, memory, and network only when they are not zero. If they are constantly zero, they are ignored. Metrics that will start having values, after Netdata is started, will be detected and charts will be automatically added to the dashboard when refreshed. Use `yes` instead of `auto` in plugin configuration sections to enable these charts permanently. You can also set the `enable zero metrics` option to `yes` in the `[global]` section which enables charts with zero metrics

175 for all internal Netdata plugins.

176

177 ### External plugins

178

215 -External plugins will have only 2 options at `netdata.conf`:

179 +External plugins will have only two options at `netdata.conf`:

180

181 | setting | default | info |

182 |:---------------:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

183 | update every | the value of `[global].update every` setting | The frequency in seconds the plugin should collect values. For more information check the [performance guide](/docs/netdata-agent/configuration/optimize-the-netdata-agents-performance.md). |

184 | command options | - | Additional command line options to pass to the plugin. |

185

222 -External plugins that need additional configuration may support a dedicated file in `/etc/netdata`. Check their

223 -documentation.

224 -

186 +External plugins that need additional configuration may support a dedicated file in `/etc/netdata`. Check their documentation.