9d06d0f0b · cryptotaxi247/netdata-1

@@ -4,13 +4,13 @@

4

5 - You can start Netdata by executing it with `/usr/sbin/netdata` (the installer will also start it).

6

7 -- You can stop Netdata by killing it with `killall netdata`.

8 - You can stop and start Netdata at any point. Netdata saves on exit its round robbin

9 - database to `/var/cache/netdata` so that it will continue from where it stopped the last time.

7 +- You can stop Netdata by killing it with `killall netdata`. You can stop and start Netdata at any point. Netdata

8 + saves on exit its round robbin database to `/var/cache/netdata` so that it will continue from where it stopped the

9 + last time.

10

11 Access to the web site, for all graphs, is by default on port `19999`, so go to:

12

13 -```

13 +```sh

14 http://127.0.0.1:19999/

15 ```

16

@@ -18,7 +18,8 @@ You can get the running config file at any time, by accessing `http://127.0.0.1:

18

19 ### Starting Netdata at boot

20

21 -In the `system` directory you can find scripts and configurations for the various distros.

21 +In the `system` directory you can find scripts and configurations for the

22 +various distros.

23

24 #### systemd

25

@@ -45,7 +46,8 @@ systemctl start netdata

46

47 #### init.d

48

48 -In the system directory you can find `netdata-lsb`. Copy it to the proper place according to your distribution documentation. For Ubuntu, this can be done via running the following commands as root.

49 +In the system directory you can find `netdata-lsb`. Copy it to the proper place according to your distribution

50 +documentation. For Ubuntu, this can be done via running the following commands as root.

51

52 ```sh

53 # copy the Netdata startup file to /etc/init.d

@@ -60,11 +62,13 @@ update-rc.d netdata defaults

62

63 #### openrc (gentoo)

64

63 -In the `system` directory you can find `netdata-openrc`. Copy it to the proper place according to your distribution documentation.

65 +In the `system` directory you can find `netdata-openrc`. Copy it to the proper

66 +place according to your distribution documentation.

67

68 #### CentOS / Red Hat Enterprise Linux

69

67 -For older versions of RHEL/CentOS that don't have systemd, an init script is included in the system directory. This can be installed by running the following commands as root.

70 +For older versions of RHEL/CentOS that don't have systemd, an init script is included in the system directory. This can

71 +be installed by running the following commands as root.

72

73 ```sh

74 # copy the Netdata startup file to /etc/init.d

@@ -77,7 +81,8 @@ chmod +x /etc/init.d/netdata

81 chkconfig --add netdata

82 ```

83

80 -_There have been some recent work on the init script, see PR <https://github.com/netdata/netdata/pull/403>_

84 +_There have been some recent work on the init script, see PR

85 +<https://github.com/netdata/netdata/pull/403>_

86

87 #### other systems

88

@@ -99,7 +104,7 @@ The program will print the supported command line parameters.

104

105 The command line options of the Netdata 1.10.0 version are the following:

106

102 -```

107 +```sh

108 ^

109 |.-. .-. .-. .-. . netdata

110 | '-' '-' '-' '-' real-time performance monitoring, done right!

@@ -188,34 +193,34 @@ Netdata uses 3 log files:

193 2. `access.log`

194 3. `debug.log`

195

191 -Any of them can be disabled by setting it to `/dev/null` or `none` in `netdata.conf`.

192 -By default `error.log` and `access.log` are enabled. `debug.log` is only enabled if

193 -debugging/tracing is also enabled (Netdata needs to be compiled with debugging enabled).

196 +Any of them can be disabled by setting it to `/dev/null` or `none` in `netdata.conf`. By default `error.log` and

197 +`access.log` are enabled. `debug.log` is only enabled if debugging/tracing is also enabled (Netdata needs to be compiled

198 +with debugging enabled).

199

200 Log files are stored in `/var/log/netdata/` by default.

201

197 -#### error.log

202 +### error.log

203

199 -The `error.log` is the `stderr` of the `netdata` daemon and all external plugins run by netdata.

204 +The `error.log` is the `stderr` of the `netdata` daemon and all external plugins

205 +run by `netdata`.

206

207 So if any process, in the Netdata process tree, writes anything to its standard error,

208 it will appear in `error.log`.

209

204 -For most Netdata programs (including standard external plugins shipped by netdata), the

205 -following lines may appear:

210 +For most Netdata programs (including standard external plugins shipped by netdata), the following lines may appear:

211

207 -| tag|description|

212 +| tag | description |

213 |:-:|:----------|

209 -| `INFO`|Something important the user should know.|

210 -| `ERROR`|Something that might disable a part of netdata. The log line includes `errno` (if it is not zero).|

211 -| `FATAL`|Something prevented a program from running. The log line includes `errno` (if it is not zero) and the program exited.|

214 +| `INFO` | Something important the user should know. |

215 +| `ERROR` | Something that might disable a part of netdata. The log line includes `errno` (if it is not zero). |

216 +| `FATAL` | Something prevented a program from running. The log line includes `errno` (if it is not zero) and the program exited. |

217

213 -So, when auto-detection of data collection fail, `ERROR` lines are logged and the relevant modules

214 -are disabled, but the program continues to run.

218 +So, when auto-detection of data collection fail, `ERROR` lines are logged and the relevant modules are disabled, but the

219 +program continues to run.

220

221 When a Netdata program cannot run at all, a `FATAL` line is logged.

222

218 -#### access.log

223 +### access.log

224

225 The `access.log` logs web requests. The format is:

226

@@ -231,21 +236,22 @@ where:

236 - `PERCENT_COMPRESSION` is the percentage of traffic saved due to compression.

237 - `PREP_TIME` is the time in milliseconds needed to prepared the response.

238 - `SENT_TIME` is the time in milliseconds needed to sent the response to the client.

234 -- `TOTAL_TIME` is the total time the request was inside Netdata (from the first byte of the request to the last byte of the response).

239 +- `TOTAL_TIME` is the total time the request was inside Netdata (from the first byte of the request to the last byte

240 + of the response).

241 - `ACTION` can be `filecopy`, `options` (used in CORS), `data` (API call).

242

237 -#### debug.log

243 +### debug.log

244

245 See [debugging](#debugging).

246

247 ## OOM Score

248

243 -Netdata runs with `OOMScore = 1000`. This means Netdata will be the first to be killed when your

244 -server runs out of memory.

249 +Netdata runs with `OOMScore = 1000`. This means Netdata will be the first to be killed when your server runs out of

250 +memory.

251

252 You can set Netdata OOMScore in `netdata.conf`, like this:

253

248 -```

254 +```conf

255 [global]

256 OOM score = 1000

257 ```

@@ -257,15 +263,13 @@ Netdata logs its OOM score when it starts:

263 2017-10-15 03:47:31: netdata INFO : Adjusted my Out-Of-Memory (OOM) score from 0 to 1000.

264 ```

265

260 -#### OOM score and systemd

266 +### OOM score and systemd

267

262 -Netdata will not be able to lower its OOM Score below zero, when it is started as the `netdata`

263 -user (systemd case).

268 +Netdata will not be able to lower its OOM Score below zero, when it is started as the `netdata` user (systemd case).

269

265 -To allow Netdata control its OOM Score in such cases, you will need to edit

266 -`netdata.service` and set:

270 +To allow Netdata control its OOM Score in such cases, you will need to edit `netdata.service` and set:

271

268 -```

272 +```sh

273 [Service]

274 # The minimum Netdata Out-Of-Memory (OOM) score.

275 # Netdata (via [global].OOM score in netdata.conf) can only increase the value set here.

@@ -276,12 +280,11 @@ OOMScoreAdjust=-1000

280

281 Run `systemctl daemon-reload` to reload these changes.

282

279 -The above, sets and OOMScore for Netdata to `-1000`, so that Netdata can increase it via

280 -`netdata.conf`.

283 +The above, sets and OOMScore for Netdata to `-1000`, so that Netdata can increase it via `netdata.conf`.

284

285 If you want to control it entirely via systemd, you can set in `netdata.conf`:

286

284 -```

287 +```conf

288 [global]

289 OOM score = keep

290 ```

@@ -290,25 +293,26 @@ Using the above, whatever OOM Score you have set at `netdata.service` will be ma

293

294 ## Netdata process scheduling policy

295

293 -By default Netdata runs with the `idle` process scheduling policy, so that it uses CPU resources, only when there is idle CPU to spare. On very busy servers (or weak servers), this can lead to gaps on the charts.

296 +By default Netdata runs with the `idle` process scheduling policy, so that it uses CPU resources, only when there is

297 +idle CPU to spare. On very busy servers (or weak servers), this can lead to gaps on the charts.

298

299 You can set Netdata scheduling policy in `netdata.conf`, like this:

300

297 -```

301 +```conf

302 [global]

303 process scheduling policy = idle

304 ```

305

306 You can use the following:

307

304 -| policy|description|

308 +| policy | description |

309 |:----:|:----------|

306 -| `idle`|use CPU only when there is spare - this is lower than nice 19 - it is the default for Netdata and it is so low that Netdata will run in "slow motion" under extreme system load, resulting in short (1-2 seconds) gaps at the charts.|

307 -| `other` or `nice`|this is the default policy for all processes under Linux. It provides dynamic priorities based on the `nice` level of each process. Check below for setting this `nice` level for netdata.|

308 -| `batch`|This policy is similar to `other` in that it schedules the thread according to its dynamic priority (based on the `nice` value). The difference is that this policy will cause the scheduler to always assume that the thread is CPU-intensive. Consequently, the scheduler will apply a small scheduling penalty with respect to wake-up behavior, so that this thread is mildly disfavored in scheduling decisions.|

309 -| `fifo`|`fifo` can be used only with static priorities higher than 0, which means that when a `fifo` threads becomes runnable, it will always immediately preempt any currently running `other`, `batch`, or `idle` thread. `fifo` is a simple scheduling algorithm without time slicing.|

310 -| `rr`|a simple enhancement of `fifo`. Everything described above for `fifo` also applies to `rr`, except that each thread is allowed to run only for a maximum time quantum.|

311 -| `keep` or `none`|do not set scheduling policy, priority or nice level - i.e. keep running with whatever it is set already (e.g. by systemd).|

310 +| `idle` | use CPU only when there is spare - this is lower than nice 19 - it is the default for Netdata and it is so low that Netdata will run in "slow motion" under extreme system load, resulting in short (1-2 seconds) gaps at the charts. |

311 +| `other` or `nice` | this is the default policy for all processes under Linux. It provides dynamic priorities based on the `nice` level of each process. Check below for setting this `nice` level for netdata. |

312 +| `batch` | This policy is similar to `other` in that it schedules the thread according to its dynamic priority (based on the `nice` value). The difference is that this policy will cause the scheduler to always assume that the thread is CPU-intensive. Consequently, the scheduler will apply a small scheduling penalty with respect to wake-up behavior, so that this thread is mildly disfavored in scheduling decisions. |

313 +| `fifo` | `fifo` can be used only with static priorities higher than 0, which means that when a `fifo` threads becomes runnable, it will always immediately preempt any currently running `other`, `batch`, or `idle` thread. `fifo` is a simple scheduling algorithm without time slicing. |

314 +| `rr` | a simple enhancement of `fifo`. Everything described above for `fifo` also applies to `rr`, except that each thread is allowed to run only for a maximum time quantum. |

315 +| `keep` or `none` | do not set scheduling policy, priority or nice level - i.e. keep running with whatever it is set already (e.g. by systemd). |

316

317 For more information see `man sched`.

318

@@ -316,29 +320,31 @@ For more information see `man sched`.

320

321 Once the policy is set to one of `rr` or `fifo`, the following will appear:

322

319 -```

323 +```conf

324 [global]

325 process scheduling priority = 0

326 ```

327

324 -These priorities are usually from 0 to 99. Higher numbers make the process more important.

328 +These priorities are usually from 0 to 99. Higher numbers make the process more

329 +important.

330

331 ### nice level for policies `other` or `batch`

332

333 When the policy is set to `other`, `nice`, or `batch`, the following will appear:

334

330 -```

335 +```conf

336 [global]

337 process nice level = 19

338 ```

339

340 ## scheduling settings and systemd

341

337 -Netdata will not be able to set its scheduling policy and priority to more important values when it is started as the `netdata` user (systemd case).

342 +Netdata will not be able to set its scheduling policy and priority to more important values when it is started as the

343 +`netdata` user (systemd case).

344

345 You can set these settings at `/etc/systemd/system/netdata.service`:

346

341 -```

347 +```sh

348 [Service]

349 # By default Netdata switches to scheduling policy idle, which makes it use CPU, only

350 # when there is spare available.

@@ -357,20 +363,23 @@ You can set these settings at `/etc/systemd/system/netdata.service`:

363

364 Run `systemctl daemon-reload` to reload these changes.

365

360 -Now, tell Netdata to keep these settings, as set by systemd, by editing `netdata.conf` and setting:

366 +Now, tell Netdata to keep these settings, as set by systemd, by editing

367 +`netdata.conf` and setting:

368

362 -```

369 +```conf

370 [global]

371 process scheduling policy = keep

372 ```

373

367 -Using the above, whatever scheduling settings you have set at `netdata.service` will be maintained by netdata.

374 +Using the above, whatever scheduling settings you have set at `netdata.service`

375 +will be maintained by netdata.

376

369 -#### Example 1: Netdata with nice -1 on non-systemd systems

377 +### Example 1: Netdata with nice -1 on non-systemd systems

378

371 -On a system that is not based on systemd, to make Netdata run with nice level -1 (a little bit higher to the default for all programs), edit `netdata.conf` and set:

379 +On a system that is not based on systemd, to make Netdata run with nice level -1 (a little bit higher to the default for

380 +all programs), edit `netdata.conf` and set:

381

373 -```

382 +```conf

383 [global]

384 process scheduling policy = other

385 process nice level = -1

@@ -384,16 +393,17 @@ sudo service netdata restart

393

394 #### Example 2: Netdata with nice -1 on systemd systems

395

387 -On a system that is based on systemd, to make Netdata run with nice level -1 (a little bit higher to the default for all programs), edit `netdata.conf` and set:

396 +On a system that is based on systemd, to make Netdata run with nice level -1 (a little bit higher to the default for all

397 +programs), edit `netdata.conf` and set:

398

389 -```

399 +```conf

400 [global]

401 process scheduling policy = keep

402 ```

403

404 edit /etc/systemd/system/netdata.service and set:

405

396 -```

406 +```sh

407 [Service]

408 CPUSchedulingPolicy=other

409 Nice=-1

@@ -408,45 +418,53 @@ sudo systemctl restart netdata

418

419 ## Virtual memory

420

411 -You may notice that netdata's virtual memory size, as reported by `ps` or `/proc/pid/status` (or even netdata's applications virtual memory chart) is unrealistically high.

421 +You may notice that netdata's virtual memory size, as reported by `ps` or `/proc/pid/status` (or even netdata's

422 +applications virtual memory chart) is unrealistically high.

423

413 -For example, it may be reported to be 150+MB, even if the resident memory size is just 25MB. Similar values may be reported for Netdata plugins too.

424 +For example, it may be reported to be 150+MB, even if the resident memory size is just 25MB. Similar values may be

425 +reported for Netdata plugins too.

426

415 -Check this for example: A Netdata installation with default settings on Ubuntu 16.04LTS. The top chart is **real memory used**, while the bottom one is **virtual memory**:

427 +Check this for example: A Netdata installation with default settings on Ubuntu

428 +16.04LTS. The top chart is **real memory used**, while the bottom one is

429 +**virtual memory**:

430

431 ![image](https://cloud.githubusercontent.com/assets/2662304/19013772/5eb7173e-87e3-11e6-8f2b-a2ccfeb06faf.png)

432

419 -**Why does this happen?**

433 +### Why does this happen?

434

421 -The system memory allocator allocates virtual memory arenas, per thread running.

422 -On Linux systems this defaults to 16MB per thread on 64 bit machines. So, if you get the

423 -difference between real and virtual memory and divide it by 16MB you will roughly get the

424 -number of threads running.

435 +The system memory allocator allocates virtual memory arenas, per thread running. On Linux systems this defaults to 16MB

436 +per thread on 64 bit machines. So, if you get the difference between real and virtual memory and divide it by 16MB you

437 +will roughly get the number of threads running.

438

426 -The system does this for speed. Having a separate memory arena for each thread, allows the

427 -threads to run in parallel in multi-core systems, without any locks between them.

439 +The system does this for speed. Having a separate memory arena for each thread, allows the threads to run in parallel in

440 +multi-core systems, without any locks between them.

441

429 -This behaviour is system specific. For example, the chart above when running Netdata on Alpine Linux (that uses **musl** instead of **glibc**) is this:

442 +This behaviour is system specific. For example, the chart above when running

443 +Netdata on Alpine Linux (that uses **musl** instead of **glibc**) is this:

444

445 ![image](https://cloud.githubusercontent.com/assets/2662304/19013807/7cf5878e-87e4-11e6-9651-082e68701eab.png)

446

433 -**Can we do anything to lower it?**

447 +### Can we do anything to lower it?

448

435 -Since Netdata already uses minimal memory allocations while it runs (i.e. it adapts its memory on start, so that while repeatedly collects data it does not do memory allocations), it already instructs the system memory allocator to minimize the memory arenas for each thread. We have also added [2 configuration options](https://github.com/netdata/netdata/blob/5645b1ee35248d94e6931b64a8688f7f0d865ec6/src/main.c#L410-L418)

436 -to allow you tweak these settings: `glibc malloc arena max for plugins` and `glibc malloc arena max for netdata`.

449 +Since Netdata already uses minimal memory allocations while it runs (i.e. it adapts its memory on start, so that while

450 +repeatedly collects data it does not do memory allocations), it already instructs the system memory allocator to

451 +minimize the memory arenas for each thread. We have also added [2 configuration

452 +options](https://github.com/netdata/netdata/blob/5645b1ee35248d94e6931b64a8688f7f0d865ec6/src/main.c#L410-L418) to allow

453 +you tweak these settings: `glibc malloc arena max for plugins` and `glibc malloc arena max for netdata`.

454

438 -However, even if we instructed the memory allocator to use just one arena, it seems it allocates an arena per thread.

455 +However, even if we instructed the memory allocator to use just one arena, it

456 +seems it allocates an arena per thread.

457

440 -Netdata also supports `jemalloc` and `tcmalloc`, however both behave exactly the same to the glibc memory allocator in this aspect.

458 +Netdata also supports `jemalloc` and `tcmalloc`, however both behave exactly the

459 +same to the glibc memory allocator in this aspect.

460

442 -**Is this a problem?**

461 +### Is this a problem?

462

463 No, it is not.

464

446 -Linux reserves real memory (physical RAM) in pages (on x86 machines pages are 4KB each).

447 -So even if the system memory allocator is allocating huge amounts of virtual memory,

448 -only the 4KB pages that are actually used are reserving physical RAM. The **real memory** chart

449 -on Netdata application section, shows the amount of physical memory these pages occupy(it

465 +Linux reserves real memory (physical RAM) in pages (on x86 machines pages are 4KB each). So even if the system memory

466 +allocator is allocating huge amounts of virtual memory, only the 4KB pages that are actually used are reserving physical

467 +RAM. The **real memory** chart on Netdata application section, shows the amount of physical memory these pages occupy(it

468 accounts the whole pages, even if parts of them are actually used).

469

470 ## Debugging

@@ -455,13 +473,19 @@ When you compile Netdata with debugging:

473

474 1. compiler optimizations for your CPU are disabled (Netdata will run somewhat slower)

475

458 -2. a lot of code is added all over netdata, to log debug messages to `/var/log/netdata/debug.log`. However, nothing is printed by default. Netdata allows you to select which sections of Netdata you want to trace. Tracing is activated via the config option `debug flags`. It accepts a hex number, to enable or disable specific sections. You can find the options supported at [log.h](../libnetdata/log/log.h). They are the `D_*` defines. The value `0xffffffffffffffff` will enable all possible debug flags.

476 +2. a lot of code is added all over netdata, to log debug messages to `/var/log/netdata/debug.log`. However, nothing is

477 + printed by default. Netdata allows you to select which sections of Netdata you want to trace. Tracing is activated

478 + via the config option `debug flags`. It accepts a hex number, to enable or disable specific sections. You can find

479 + the options supported at [log.h](../libnetdata/log/log.h). They are the `D_*` defines. The value

480 + `0xffffffffffffffff` will enable all possible debug flags.

481

460 -Once Netdata is compiled with debugging and tracing is enabled for a few sections, the file `/var/log/netdata/debug.log` will contain the messages.

482 +Once Netdata is compiled with debugging and tracing is enabled for a few sections, the file `/var/log/netdata/debug.log`

483 +will contain the messages.

484

462 -> Do not forget to disable tracing (`debug flags = 0`) when you are done tracing. The file `debug.log` can grow too fast.

485 +> Do not forget to disable tracing (`debug flags = 0`) when you are done tracing. The file `debug.log` can grow too

486 +> fast.

487

464 -#### compiling Netdata with debugging

488 +### compiling Netdata with debugging

489

490 To compile Netdata with debugging, use this:

491

@@ -473,13 +497,17 @@ cd /usr/src/netdata.git

497 CFLAGS="-O1 -ggdb -DNETDATA_INTERNAL_CHECKS=1" ./netdata-installer.sh

498 ```

499

476 -The above will compile and install Netdata with debugging info embedded. You can now use `debug flags` to set the section(s) you need to trace.

500 +The above will compile and install Netdata with debugging info embedded. You can now use `debug flags` to set the

501 +section(s) you need to trace.

502

478 -#### debugging crashes

503 +### debugging crashes

504

480 -We have made the most to make Netdata crash free. If however, Netdata crashes on your system, it would be very helpful to provide stack traces of the crash. Without them, is will be almost impossible to find the issue (the code base is quite large to find such an issue by just objerving it).

505 +We have made the most to make Netdata crash free. If however, Netdata crashes on your system, it would be very helpful

506 +to provide stack traces of the crash. Without them, is will be almost impossible to find the issue (the code base is

507 +quite large to find such an issue by just objerving it).

508

482 -To provide stack traces, **you need to have Netdata compiled with debugging**. There is no need to enable any tracing (`debug flags`).

509 +To provide stack traces, **you need to have Netdata compiled with debugging**. There is no need to enable any tracing

510 +(`debug flags`).

511

512 Then you need to be in one of the following 2 cases:

513

@@ -487,9 +515,10 @@ Then you need to be in one of the following 2 cases:

515

516 2. you can reproduce the crash

517

490 -If you are not on these cases, you need to find a way to be (i.e. if your system does not produce core dumps, check your distro documentation to enable them).

518 +If you are not on these cases, you need to find a way to be (i.e. if your system does not produce core dumps, check your

519 +distro documentation to enable them).

520

492 -#### Netdata crashes and you have a core dump

521 +### Netdata crashes and you have a core dump

522

523 > you need to have Netdata compiled with debugging info for this to work (check above)

524

@@ -499,7 +528,7 @@ Run the following command and post the output on a github issue.

528 gdb $(which netdata) /path/to/core/dump

529 ```

530

502 -#### you can reproduce a Netdata crash on your system

531 +### you can reproduce a Netdata crash on your system

532

533 > you need to have Netdata compiled with debugging info for this to work (check above)

534

@@ -509,6 +538,7 @@ Install the package `valgrind` and run:

538 valgrind $(which netdata) -D

539 ```

540

512 -Netdata will start and it will be a lot slower. Now reproduce the crash and `valgrind` will dump on your console the stack trace. Open a new github issue and post the output.

541 +Netdata will start and it will be a lot slower. Now reproduce the crash and `valgrind` will dump on your console the

542 +stack trace. Open a new github issue and post the output.

543

544 [![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fdaemon%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)

Fix remark warnings for Daemon README (#6920)