src dir docs pass (#18670)
Fotis Voutsas committed
Oct 3, 2024 at 15:38 UTC
0213967d71a33cf74ce13aed72d68a1ca0b48140
28 files changed
+733
-865
src/aclk/README.md
+5
-4
@@ -4,13 +4,13 @@ The Agent-Cloud link (ACLK) is the mechanism responsible for securely connecting
4
through Netdata Cloud. The ACLK establishes an outgoing secure WebSocket (WSS) connection to Netdata Cloud on port
5
`443`. The ACLK is encrypted, safe, and _is only established if you connect your node_.
6
7
-The Cloud App lives at app.netdata.cloud which currently resolves to the following list of IPs:
7
+The Cloud App lives at app.netdata.cloud which currently resolves to the following list of IPs:
8
9
- 54.198.178.11
10
- 44.207.131.212
11
-- 44.196.50.41
11
+- 44.196.50.41
12
13
-> ### Caution
13
+> **Caution**
14
>
15
>This list of IPs can change without notice, we strongly advise you to whitelist following domains `app.netdata.cloud`, `mqtt.netdata.cloud`, if this is not an option in your case always verify the current domain resolution (e.g via the `host` command).
16
@@ -34,7 +34,8 @@ If your Agent needs to use a proxy to access the internet, you must [set up a pr
34
connecting to cloud](/src/claim/README.md).
35
36
You can configure following keys in the `netdata.conf` section `[cloud]`:
37
-```
37
+
38
+```text
39
[cloud]
40
statistics = yes
41
query thread count = 2
src/claim/README.md
+3
-2
@@ -102,8 +102,9 @@ cd /var/lib/netdata # Replace with your Netdata library directory, if not /var
102
sudo rm -rf cloud.d/
103
```
104
105
-> IMPORTANT:<br/>
106
-> Keep in mind that the Agent will be **re-claimed automatically** if the environment variables or `claim.conf` exist when the agent is restarted.
105
+> **IMPORTANT**
106
+>
107
+> Keep in mind that the Agent will be **re-claimed automatically** if the environment variables or `claim.conf` exist when the agent is restarted.
108
109
This node no longer has access to the credentials it was used when connecting to Netdata Cloud via the ACLK. You will
110
still be able to see this node in your Rooms in an **unreachable** state.
src/cli/README.md
+1
-3
@@ -18,9 +18,7 @@ Available commands:
18
| `ping` | Checks the Agent's status. If the Agent is alive, it exits with status code 0 and prints 'pong' to standard output. Exits with status code 255 otherwise. |
19
| `aclk-state [json]` | Return the current state of ACLK and Cloud connection. Optionally in JSON. |
20
| `dumpconfig` | Display the current netdata.conf configuration. |
21
-| `remove-stale-node <node_id \| machine_guid \| hostname \| ALL_NODES>` | Unregisters a stale child Node, removing it from the parent Node's UI and Netdata Cloud. This is useful for ephemeral Nodes that may stop streaming and remain visible as stale. |
21
+| `remove-stale-node <node_id \| machine_guid \| hostname \| ALL_NODES>` | Un-registers a stale child Node, removing it from the parent Node's UI and Netdata Cloud. This is useful for ephemeral Nodes that may stop streaming and remain visible as stale. |
22
| `version` | Display the Netdata Agent version. |
23
24
See also the Netdata daemon [command line options](/src/daemon/README.md#command-line-options).
25
-
26
-
src/collectors/README.md
+16
-16
@@ -7,7 +7,7 @@ Netdata can immediately collect metrics from these endpoints thanks to 300+ **co
7
when you [install Netdata](/packaging/installer/README.md).
8
9
All collectors are **installed by default** with every installation of Netdata. You do not need to install
10
-collectors manually to collect metrics from new sources.
10
+collectors manually to collect metrics from new sources.
11
See how you can [monitor anything with Netdata](/src/collectors/COLLECTORS.md).
12
13
Upon startup, Netdata will **auto-detect** any application or service that has a collector, as long as both the collector
@@ -18,45 +18,45 @@ our [collectors' configuration reference](/src/collectors/REFERENCE.md).
18
19
Every collector has two primary jobs:
20
21
-- Look for exposed metrics at a pre- or user-defined endpoint.
22
-- Gather exposed metrics and use additional logic to build meaningful, interactive visualizations.
21
+- Look for exposed metrics at a pre- or user-defined endpoint.
22
+- Gather exposed metrics and use additional logic to build meaningful, interactive visualizations.
23
24
If the collector finds compatible metrics exposed on the configured endpoint, it begins a per-second collection job. The
25
-Netdata Agent gathers these metrics, sends them to the
25
+Netdata Agent gathers these metrics, sends them to the
26
[database engine for storage](/docs/netdata-agent/configuration/optimizing-metrics-database/change-metrics-storage.md)
27
-, and immediately
28
-[visualizes them meaningfully](/docs/dashboards-and-charts/netdata-charts.md)
27
+, and immediately
28
+[visualizes them meaningfully](/docs/dashboards-and-charts/netdata-charts.md)
29
on dashboards.
30
31
Each collector comes with a pre-defined configuration that matches the default setup for that application. This endpoint
32
-can be a URL and port, a socket, a file, a web page, and more. The endpoint is user-configurable, as are many other
32
+can be a URL and port, a socket, a file, a web page, and more. The endpoint is user-configurable, as are many other
33
specifics of what a given collector does.
34
35
## Collector architecture and terminology
36
37
-- **Collectors** are the processes/programs that actually gather metrics from various sources.
37
+- **Collectors** are the processes/programs that actually gather metrics from various sources.
38
39
-- **Plugins** help manage all the independent data collection processes in a variety of programming languages, based on
39
+- **Plugins** help manage all the independent data collection processes in a variety of programming languages, based on
40
their purpose and performance requirements. There are three types of plugins:
41
42
- - **Internal** plugins organize collectors that gather metrics from `/proc`, `/sys` and other Linux kernel sources.
42
+ - **Internal** plugins organize collectors that gather metrics from `/proc`, `/sys` and other Linux kernel sources.
43
They are written in `C`, and run as threads within the Netdata daemon.
44
45
- - **External** plugins organize collectors that gather metrics from external processes, such as a MySQL database or
45
+ - **External** plugins organize collectors that gather metrics from external processes, such as a MySQL database or
46
Nginx web server. They can be written in any language, and the `netdata` daemon spawns them as long-running
47
independent processes. They communicate with the daemon via pipes. All external plugins are managed by
48
[plugins.d](/src/plugins.d/README.md), which provides additional management options.
49
50
-- **Orchestrators** are external plugins that run and manage one or more modules. They run as independent processes.
50
+- **Orchestrators** are external plugins that run and manage one or more modules. They run as independent processes.
51
The Go orchestrator is in active development.
52
53
- - [go.d.plugin](/src/go/plugin/go.d/README.md): An orchestrator for data
53
+ - [go.d.plugin](/src/go/plugin/go.d/README.md): An orchestrator for data
54
collection modules written in `go`.
55
56
- - [python.d.plugin](/src/collectors/python.d.plugin/README.md):
56
+ - [python.d.plugin](/src/collectors/python.d.plugin/README.md):
57
An orchestrator for data collection modules written in `python` v2/v3.
58
59
- - [charts.d.plugin](/src/collectors/charts.d.plugin/README.md):
59
+ - [charts.d.plugin](/src/collectors/charts.d.plugin/README.md):
60
An orchestrator for data collection modules written in`bash` v4+.
61
62
-- **Modules** are the individual programs controlled by an orchestrator to collect data from a specific application, or type of endpoint.
62
+- **Modules** are the individual programs controlled by an orchestrator to collect data from a specific application, or type of endpoint.
src/collectors/REFERENCE.md
+43
-54
@@ -1,32 +1,23 @@
1
-<!--
2
-title: "Collectors configuration reference"
3
-custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/REFERENCE.md"
4
-sidebar_label: "Collectors configuration"
5
-learn_status: "Published"
6
-learn_topic_type: "Tasks"
7
-learn_rel_path: "Configuration"
8
--->
9
-
1
# Collectors configuration reference
2
12
-The list of supported collectors can be found in [the documentation](/src/collectors/COLLECTORS.md),
13
-and on [our website](https://www.netdata.cloud/integrations). The documentation of each collector provides all the
14
-necessary configuration options and prerequisites for that collector. In most cases, either the charts are automatically generated
3
+The list of supported collectors can be found in [the documentation](/src/collectors/COLLECTORS.md),
4
+and on [our website](https://www.netdata.cloud/integrations). The documentation of each collector provides all the
5
+necessary configuration options and prerequisites for that collector. In most cases, either the charts are automatically generated
6
without any configuration, or you just fulfil those prerequisites and [configure the collector](#configure-a-collector).
7
17
-If the application you are interested in monitoring is not listed in our integrations, the collectors list includes
18
-the available options to
8
+If the application you are interested in monitoring is not listed in our integrations, the collectors list includes
9
+the available options to
10
[add your application to Netdata](https://github.com/netdata/netdata/edit/master/src/collectors/COLLECTORS.md#add-your-application-to-netdata).
11
21
-If we do support your collector but the charts described in the documentation don't appear on your dashboard, the reason will
12
+If we do support your collector but the charts described in the documentation don't appear on your dashboard, the reason will
13
be one of the following:
14
24
-- The entire data collection plugin is disabled by default. Read how to [enable and disable plugins](#enable-and-disable-plugins)
15
+- The entire data collection plugin is disabled by default. Read how to [enable and disable plugins](#enable-and-disable-plugins)
16
26
-- The data collection plugin is enabled, but a specific data collection module is disabled. Read how to
27
- [enable and disable a specific collection module](#enable-and-disable-a-specific-collection-module).
17
+- The data collection plugin is enabled, but a specific data collection module is disabled. Read how to
18
+ [enable and disable a specific collection module](#enable-and-disable-a-specific-collection-module).
19
29
-- Autodetection failed. Read how to [configure](#configure-a-collector) and [troubleshoot](#troubleshoot-a-collector) a collector.
20
+- Autodetection failed. Read how to [configure](#configure-a-collector) and [troubleshoot](#troubleshoot-a-collector) a collector.
21
22
## Enable and disable plugins
23
@@ -36,26 +27,26 @@ This section features a list of Netdata's plugins, with a boolean setting to ena
27
28
```conf
29
[plugins]
39
- # timex = yes
40
- # idlejitter = yes
41
- # netdata monitoring = yes
42
- # tc = yes
43
- # diskspace = yes
44
- # proc = yes
45
- # cgroups = yes
46
- # enable running new plugins = yes
47
- # check for new plugins every = 60
48
- # slabinfo = no
49
- # python.d = yes
50
- # perf = yes
51
- # ioping = yes
52
- # fping = yes
53
- # nfacct = yes
54
- # go.d = yes
55
- # apps = yes
56
- # ebpf = yes
57
- # charts.d = yes
58
- # statsd = yes
30
+ # timex = yes
31
+ # idlejitter = yes
32
+ # netdata monitoring = yes
33
+ # tc = yes
34
+ # diskspace = yes
35
+ # proc = yes
36
+ # cgroups = yes
37
+ # enable running new plugins = yes
38
+ # check for new plugins every = 60
39
+ # slabinfo = no
40
+ # python.d = yes
41
+ # perf = yes
42
+ # ioping = yes
43
+ # fping = yes
44
+ # nfacct = yes
45
+ # go.d = yes
46
+ # apps = yes
47
+ # ebpf = yes
48
+ # charts.d = yes
49
+ # statsd = yes
50
```
51
52
By default, most plugins are enabled, so you don't need to enable them explicitly to use their collectors. To enable or
@@ -63,11 +54,11 @@ disable any specific plugin, remove the comment (`#`) and change the boolean set
54
55
## Enable and disable a specific collection module
56
66
-You can enable/disable of the collection modules supported by `go.d`, `python.d` or `charts.d` individually, using the
67
-configuration file of that orchestrator. For example, you can change the behavior of the Go orchestrator, or any of its
57
+You can enable/disable of the collection modules supported by `go.d`, `python.d` or `charts.d` individually, using the
58
+configuration file of that orchestrator. For example, you can change the behavior of the Go orchestrator, or any of its
59
collectors, by editing `go.d.conf`.
60
70
-Use `edit-config` from your [Netdata config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory)
61
+Use `edit-config` from your [Netdata config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory)
62
to open the orchestrator primary configuration file:
63
64
```bash
@@ -79,20 +70,19 @@ Within this file, you can either disable the orchestrator entirely (`enabled: ye
70
enable/disable it with `yes` and `no` settings. Uncomment any line you change to ensure the Netdata daemon reads it on
71
start.
72
82
-After you make your changes, restart the Agent with `sudo systemctl restart netdata`, or the [appropriate
83
-method](/packaging/installer/README.md#maintaining-a-netdata-agent-installation) for your system.
73
+After you make your changes, restart the Agent with the [appropriate method](/docs/netdata-agent/start-stop-restart.md) for your system.
74
75
## Configure a collector
76
77
Most collector modules come with **auto-detection**, configured to work out-of-the-box on popular operating systems with
88
-the default settings.
78
+the default settings.
79
80
However, there are cases that auto-detection fails. Usually, the reason is that the applications to be monitored do not
81
allow Netdata to connect. In most of the cases, allowing the user `netdata` from `localhost` to connect and collect
82
metrics, will automatically enable data collection for the application in question (it will require a Netdata restart).
83
84
When Netdata starts up, each collector searches for exposed metrics on the default endpoint established by that service
95
-or application's standard installation procedure. For example,
85
+or application's standard installation procedure. For example,
86
the [Nginx collector](/src/go/plugin/go.d/modules/nginx/README.md) searches at
87
`http://127.0.0.1/stub_status` for exposed metrics in the correct format. If an Nginx web server is running and exposes
88
metrics on that endpoint, the collector begins gathering them.
@@ -100,12 +90,12 @@ metrics on that endpoint, the collector begins gathering them.
90
However, not every node or infrastructure uses standard ports, paths, files, or naming conventions. You may need to
91
enable or configure a collector to gather all available metrics from your systems, containers, or applications.
92
103
-First, [find the collector](/src/collectors/COLLECTORS.md) you want to edit
104
-and open its documentation. Some software has collectors written in multiple languages. In these cases, you should always
93
+First, [find the collector](/src/collectors/COLLECTORS.md) you want to edit
94
+and open its documentation. Some software has collectors written in multiple languages. In these cases, you should always
95
pick the collector written in Go.
96
107
-Use `edit-config` from your
108
-[Netdata config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory)
97
+Use `edit-config` from your
98
+[Netdata config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory)
99
to open a collector's configuration file. For example, edit the Nginx collector with the following:
100
101
```bash
@@ -117,8 +107,7 @@ according to your needs. In addition, every collector's documentation shows the
107
configure that collector. Uncomment any line you change to ensure the collector's orchestrator or the Netdata daemon
108
read it on start.
109
120
-After you make your changes, restart the Agent with `sudo systemctl restart netdata`, or the [appropriate
121
-method](/packaging/installer/README.md#maintaining-a-netdata-agent-installation) for your system.
110
+After you make your changes, restart the Agent with the [appropriate method](/docs/netdata-agent/start-stop-restart.md) for your system.
111
112
## Troubleshoot a collector
113
@@ -131,7 +120,7 @@ cd /usr/libexec/netdata/plugins.d/
120
sudo su -s /bin/bash netdata
121
```
122
134
-The next step is based on the collector's orchestrator.
123
+The next step is based on the collector's orchestrator.
124
125
```bash
126
# Go orchestrator (go.d.plugin)
@@ -145,5 +134,5 @@ The next step is based on the collector's orchestrator.
134
```
135
136
The output from the relevant command will provide valuable troubleshooting information. If you can't figure out how to
148
-enable the collector using the details from this output, feel free to [join our Discord server](https://discord.com/invite/2mEmfW735j),
137
+enable the collector using the details from this output, feel free to [join our Discord server](https://discord.com/invite/2mEmfW735j),
138
to get help from our experts.
src/collectors/apps.plugin/README.md
+79
-88
@@ -1,12 +1,3 @@
1
-<!--
2
-title: "Application monitoring (apps.plugin)"
3
-sidebar_label: "Application monitoring "
4
-custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/apps.plugin/README.md"
5
-learn_status: "Published"
6
-learn_topic_type: "References"
7
-learn_rel_path: "Integrations/Monitor/System metrics"
8
--->
9
-
1
# Applications monitoring (apps.plugin)
2
3
`apps.plugin` monitors the resources utilization of all processes running.
@@ -16,21 +7,21 @@ learn_rel_path: "Integrations/Monitor/System metrics"
7
`apps.plugin` aggregates processes in three distinct ways to provide a more insightful
8
breakdown of resource utilization:
9
19
- - **Tree** or **Category**: Grouped by their position in the process tree.
10
+- **Tree** or **Category**: Grouped by their position in the process tree.
11
This is customizable and allows aggregation by process managers and individual
12
processes of interest. Allows also renaming the processes for presentation purposes.
22
-
23
- - **User**: Grouped by the effective user (UID) under which the processes run.
24
-
25
- - **Group**: Grouped by the effective group (GID) under which the processes run.
13
27
- ## Short-Lived Process Handling
14
+- **User**: Grouped by the effective user (UID) under which the processes run.
15
+
16
+- **Group**: Grouped by the effective group (GID) under which the processes run.
17
+
18
+## Short-Lived Process Handling
19
20
`apps.plugin` accounts for resource utilization of both running and exited processes,
21
capturing the impact of processes that spawn short-lived subprocesses, such as shell
22
scripts that fork hundreds or thousands of times per second. So, although processes
23
may spawn short lived sub-processes, `apps.plugin` will aggregate their resources
33
-utilization providing a holistic view of how resources are shared among the processes.
24
+utilization providing a holistic view of how resources are shared among the processes.
25
26
## Charts sections
27
@@ -40,7 +31,7 @@ Each type of aggregation is presented as a different section on the dashboard.
31
### Custom Process Groups (Apps)
32
33
In this section, apps.plugin summarizes the resources consumed by all processes, grouped based
43
-on the groups provided in `/etc/netdata/apps_groups.conf`. You can edit this file using our [`edit-config`](docs/netdata-agent/configuration/README.md) script.
34
+on the groups provided in `/etc/netdata/apps_groups.conf`. You can edit this file using our [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) script.
35
36
For this section, `apps.plugin` builds a process tree (much like `ps fax` does in Linux), and groups
37
processes together (evaluating both child and parent processes) so that the result is always a list with
@@ -63,46 +54,46 @@ effective user group under which each process runs.
54
55
`apps.plugin` provides charts for 3 sections:
56
66
-1. Per application charts as **Applications** at Netdata dashboards
67
-2. Per user charts as **Users** at Netdata dashboards
68
-3. Per user group charts as **User Groups** at Netdata dashboards
57
+1. Per application charts as **Applications** at Netdata dashboards
58
+2. Per user charts as **Users** at Netdata dashboards
59
+3. Per user group charts as **User Groups** at Netdata dashboards
60
61
Each of these sections provides the same number of charts:
62
72
-- CPU utilization (`apps.cpu`)
73
- - Total CPU usage
74
- - User/system CPU usage (`apps.cpu_user`/`apps.cpu_system`)
75
-- Disk I/O
76
- - Physical reads/writes (`apps.preads`/`apps.pwrites`)
77
- - Logical reads/writes (`apps.lreads`/`apps.lwrites`)
78
- - Open unique files (if a file is found open multiple times, it is counted just once, `apps.files`)
79
-- Memory
80
- - Real Memory Used (non-shared, `apps.mem`)
81
- - Virtual Memory Allocated (`apps.vmem`)
82
- - Minor page faults (i.e. memory activity, `apps.minor_faults`)
83
-- Processes
84
- - Threads running (`apps.threads`)
85
- - Processes running (`apps.processes`)
86
- - Carried over uptime (since the last Netdata Agent restart, `apps.uptime`)
87
- - Minimum uptime (`apps.uptime_min`)
88
- - Average uptime (`apps.uptime_average`)
89
- - Maximum uptime (`apps.uptime_max`)
90
- - Pipes open (`apps.pipes`)
91
-- Swap memory
92
- - Swap memory used (`apps.swap`)
93
- - Major page faults (i.e. swap activity, `apps.major_faults`)
94
-- Network
95
- - Sockets open (`apps.sockets`)
96
-
63
+- CPU utilization (`apps.cpu`)
64
+ - Total CPU usage
65
+ - User/system CPU usage (`apps.cpu_user`/`apps.cpu_system`)
66
+- Disk I/O
67
+ - Physical reads/writes (`apps.preads`/`apps.pwrites`)
68
+ - Logical reads/writes (`apps.lreads`/`apps.lwrites`)
69
+ - Open unique files (if a file is found open multiple times, it is counted just once, `apps.files`)
70
+- Memory
71
+ - Real Memory Used (non-shared, `apps.mem`)
72
+ - Virtual Memory Allocated (`apps.vmem`)
73
+ - Minor page faults (i.e. memory activity, `apps.minor_faults`)
74
+- Processes
75
+ - Threads running (`apps.threads`)
76
+ - Processes running (`apps.processes`)
77
+ - Carried over uptime (since the last Netdata Agent restart, `apps.uptime`)
78
+ - Minimum uptime (`apps.uptime_min`)
79
+ - Average uptime (`apps.uptime_average`)
80
+ - Maximum uptime (`apps.uptime_max`)
81
+ - Pipes open (`apps.pipes`)
82
+- Swap memory
83
+ - Swap memory used (`apps.swap`)
84
+ - Major page faults (i.e. swap activity, `apps.major_faults`)
85
+- Network
86
+ - Sockets open (`apps.sockets`)
87
+
88
In addition, if the [eBPF collector](/src/collectors/ebpf.plugin/README.md) is running, your dashboard will also show an
89
additional [list of charts](/src/collectors/ebpf.plugin/README.md#integration-with-appsplugin) using low-level Linux
90
metrics.
91
92
The above are reported:
93
103
-- For **Applications** per target configured.
104
-- For **Users** per username or UID (when the username is not available).
105
-- For **User Groups** per group name or GID (when group name is not available).
94
+- For **Applications** per target configured.
95
+- For **Users** per username or UID (when the username is not available).
96
+- For **User Groups** per group name or GID (when group name is not available).
97
98
## Performance
99
@@ -119,10 +110,10 @@ In such cases, you many need to lower its data collection frequency.
110
111
To do this, edit `/etc/netdata/netdata.conf` and find this section:
112
122
-```
113
+```txt
114
[plugin:apps]
124
- # update every = 1
125
- # command options =
115
+ # update every = 1
116
+ # command options =
117
```
118
119
Uncomment the line `update every` and set it to a higher number. If you just set it to `2`,
@@ -130,7 +121,7 @@ its CPU resources will be cut in half, and data collection will be once every 2
121
122
## Configuration
123
133
-The configuration file is `/etc/netdata/apps_groups.conf`. You can edit this file using our [`edit-config`](docs/netdata-agent/configuration/README.md) script.
124
+The configuration file is `/etc/netdata/apps_groups.conf`. You can edit this file using our [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) script.
125
126
### Configuring process managers
127
@@ -140,7 +131,7 @@ consider all their sub-processes important to monitor.
131
132
Process managers are configured in `apps_groups.conf` with the prefix `managers:`, like this:
133
143
-```
134
+```txt
135
managers: process1 process2 process3
136
```
137
@@ -164,8 +155,8 @@ For each process given, all of its sub-processes will be grouped, not just the m
155
156
The process names are the ones returned by:
157
167
-- **comm**: `ps -e` or `cat /proc/{PID}/stat`
168
-- **cmdline**: in case of substring mode (see below): `/proc/{PID}/cmdline`
158
+- **comm**: `ps -e` or `cat /proc/{PID}/stat`
159
+- **cmdline**: in case of substring mode (see below): `/proc/{PID}/cmdline`
160
161
On Linux **comm** is limited to just a few characters. `apps.plugin` attempts to find the entire
162
**comm** name by looking for it at the **cmdline**. When this is successful, the entire process name
@@ -176,12 +167,12 @@ example: `'Plex Media Serv'` or `"my other process"`.
167
168
You can add asterisks (`*`) to provide a pattern:
169
179
-- `*name` _suffix_ mode: will match a **comm** ending with `name`.
180
-- `name*` _prefix_ mode: will match a **comm** beginning with `name`.
181
-- `*name*` _substring_ mode: will search for `name` in **cmdline**.
170
+- `*name` _suffix_ mode: will match a **comm** ending with `name`.
171
+- `name*` _prefix_ mode: will match a **comm** beginning with `name`.
172
+- `*name*` _substring_ mode: will search for `name` in **cmdline**.
173
174
Asterisks may appear in the middle of `name` (like `na*me`), without affecting what is being
184
-matched (**comm** or **cmdline**).
175
+matched (**comm** or **cmdline**).
176
177
To add processes with single quotes, enclose them in double quotes: `"process with this ' single quote"`
178
@@ -194,7 +185,7 @@ There are a few command line options you can pass to `apps.plugin`. The list of
185
options can be acquired with the `--help` flag. The options can be set in the `netdata.conf` using the [`edit-config` script](/docs/netdata-agent/configuration/README.md).
186
For example, to disable user and user group charts you would set:
187
197
-```
188
+```txt
189
[plugin:apps]
190
command options = without-users without-groups
191
```
@@ -246,7 +237,7 @@ but it will not be able to collect all the information.
237
238
You can create badges that you can embed anywhere you like, with URLs like this:
239
249
-```
240
+```txt
241
https://your.netdata.ip:19999/api/v1/badge.svg?chart=apps.processes&dimensions=myapp&value_color=green%3E0%7Cred
242
```
243
@@ -259,23 +250,23 @@ Here is an example for the process group `sql` at `https://registry.my-netdata.i
250
Netdata is able to give you a lot more badges for your app.
251
Examples below for process group `sql`:
252
262
-- CPU usage: 
263
-- Disk Physical Reads 
264
-- Disk Physical Writes 
265
-- Disk Logical Reads 
266
-- Disk Logical Writes 
267
-- Open Files 
268
-- Real Memory 
269
-- Virtual Memory 
270
-- Swap Memory 
271
-- Minor Page Faults 
272
-- Processes 
273
-- Threads 
274
-- Major Faults (swap activity) 
275
-- Open Pipes 
276
-- Open Sockets 
277
-
278
-For more information about badges check [Generating Badges](/src/web/api/v2/api_v3_badge/README.md)
253
+- CPU usage: 
254
+- Disk Physical Reads 
255
+- Disk Physical Writes 
256
+- Disk Logical Reads 
257
+- Disk Logical Writes 
258
+- Open Files 
259
+- Real Memory 
260
+- Virtual Memory 
261
+- Swap Memory 
262
+- Minor Page Faults 
263
+- Processes 
264
+- Threads 
265
+- Major Faults (swap activity) 
266
+- Open Pipes 
267
+- Open Sockets 
268
+
269
+<!-- For more information about badges check [Generating Badges](/src/web/api/v2/api_v3_badge/README.md) -->
270
271
## Comparison with console tools
272
@@ -302,7 +293,7 @@ If you check the total system CPU utilization, it says there is no idle CPU at a
293
fails to provide a breakdown of the CPU consumption in the system. The sum of the CPU utilization
294
of all processes reported by `top`, is 15.6%.
295
305
-```
296
+```txt
297
top - 18:46:28 up 3 days, 20:14, 2 users, load average: 0.22, 0.05, 0.02
298
Tasks: 76 total, 2 running, 74 sleeping, 0 stopped, 0 zombie
299
%Cpu(s): 32.8 us, 65.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 1.3 hi, 0.3 si, 0.0 st
@@ -322,7 +313,7 @@ KiB Swap: 0 total, 0 free, 0 used. 753712 avail Mem
313
314
Exactly like `top`, `htop` is providing an incomplete breakdown of the system CPU utilization.
315
325
-```
316
+```bash
317
CPU[||||||||||||||||||||||||100.0%] Tasks: 27, 11 thr; 2 running
318
Mem[||||||||||||||||||||85.4M/993M] Load average: 1.16 0.88 0.90
319
Swp[ 0K/0K] Uptime: 3 days, 21:37:03
@@ -332,7 +323,7 @@ Exactly like `top`, `htop` is providing an incomplete breakdown of the system CP
323
7024 netdata 20 0 9544 2480 1744 S 0.7 0.2 0:00.88 /usr/libexec/netd
324
7009 netdata 20 0 138M 21016 2712 S 0.7 2.1 0:00.89 /usr/sbin/netdata
325
7012 netdata 20 0 138M 21016 2712 S 0.0 2.1 0:00.31 /usr/sbin/netdata
335
- 563 root 20 0 308M 202M 202M S 0.0 20.4 1:00.81 /usr/lib/systemd/
326
+ 563 root 20 0 308M 202M 202M S 0.0 20.4 1:00.81 /usr/lib/systemd/
327
7019 netdata 20 0 138M 21016 2712 S 0.0 2.1 0:00.14 /usr/sbin/netdata
328
```
329
@@ -340,7 +331,7 @@ Exactly like `top`, `htop` is providing an incomplete breakdown of the system CP
331
332
`atop` also fails to break down CPU usage.
333
343
-```
334
+```bash
335
ATOP - localhost 2016/12/10 20:11:27 ----------- 10s elapsed
336
PRC | sys 1.13s | user 0.43s | #proc 75 | #zombie 0 | #exit 5383 |
337
CPU | sys 67% | user 31% | irq 2% | idle 0% | wait 0% |
@@ -356,7 +347,7 @@ NET | eth0 ---- | pcki 16 | pcko 15 | si 1 Kbps | so 4 Kbps |
347
12789 0.98s 0.40s 0K 0K 0K 336K -- - S 14% bash
348
9 0.08s 0.00s 0K 0K 0K 0K -- - S 1% rcuos/0
349
7024 0.03s 0.00s 0K 0K 0K 0K -- - S 0% apps.plugin
359
- 7009 0.01s 0.01s 0K 0K 0K 4K -- - S 0% netdata
350
+ 7009 0.01s 0.01s 0K 0K 0K 4K -- - S 0% netdata
351
```
352
353
### glances
@@ -366,7 +357,7 @@ per process utilization.
357
358
Note also, that being a `python` program, `glances` uses 1.6% CPU while it runs.
359
369
-```
360
+```bash
361
localhost Uptime: 3 days, 21:42:00
362
363
CPU [100.0%] CPU 100.0% MEM 23.7% SWAP 0.0% LOAD 1-core
@@ -388,8 +379,8 @@ FILE SYS Used Total 0.3 2.1 7009 netdata 0 S /usr/sbin/netdata
379
380
### why does this happen?
381
391
-All the console tools report usage based on the processes found running *at the moment they
392
-examine the process tree*. So, they see just one `ls` command, which is actually very quick
382
+All the console tools report usage based on the processes found running _at the moment they
383
+examine the process tree_. So, they see just one `ls` command, which is actually very quick
384
with minor CPU utilization. But the shell, is spawning hundreds of them, one after another
385
(much like shell scripts do).
386
@@ -398,12 +389,12 @@ with minor CPU utilization. But the shell, is spawning hundreds of them, one aft
389
The total CPU utilization of the system:
390
391
401
-<br/>***Figure 1**: The system overview section at Netdata, just a few seconds after the command was run*
392
+<br/>_**Figure 1**: The system overview section at Netdata, just a few seconds after the command was run_
393
394
And at the applications `apps.plugin` breaks down CPU usage per application:
395
396
406
-<br/>***Figure 2**: The Applications section at Netdata, just a few seconds after the command was run*
397
+<br/>_**Figure 2**: The Applications section at Netdata, just a few seconds after the command was run_
398
399
So, the `ssh` session is using 95% CPU time.
400
src/collectors/cgroups.plugin/README.md
-9
@@ -1,12 +1,3 @@
1
-<!--
2
-title: "Monitor Cgroups (cgroups.plugin)"
3
-custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/cgroups.plugin/README.md"
4
-sidebar_label: "Monitor Cgroups"
5
-learn_status: "Published"
6
-learn_topic_type: "References"
7
-learn_rel_path: "Integrations/Monitor/Virtualized environments/Containers"
8
--->
9
-
1
# Monitor Cgroups (cgroups.plugin)
2
3
You can monitor containers and virtual machines using **cgroups**.
src/collectors/charts.d.plugin/README.md
+1
-2
@@ -9,7 +9,6 @@
9
10
To better understand the guidelines and the API behind our External plugins, please have a look at the [Introduction to External plugins](/src/plugins.d/README.md) prior to reading this page.
11
12
-
12
`charts.d.plugin` has been designed so that the actual script that will do data collection will be permanently in
13
memory, collecting data with as little overheads as possible
14
(i.e. initialize once, repeatedly collect values with minimal overhead).
@@ -121,7 +120,7 @@ Using the above, if the command `mysql` is not available in the system, the `mys
120
`fixid()` will get a string and return a properly formatted id for a chart or dimension.
121
122
This is an expensive function that should not be used in `X_update()`.
124
-You can keep the generated id in a BASH associative array to have the values availables in `X_update()`, like this:
123
+You can keep the generated id in a BASH associative array to have the values available in `X_update()`, like this:
124
125
```sh
126
declare -A X_ids=()
src/collectors/ebpf.plugin/README.md
+220
-230
@@ -1,16 +1,6 @@
1
-<!--
2
-title: "Kernel traces/metrics (eBPF) monitoring with Netdata"
3
-description: "Use Netdata's extended Berkeley Packet Filter (eBPF) collector to monitor kernel-level metrics about yourcomplex applications with per-second granularity."
4
-custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/ebpf.plugin/README.md"
5
-sidebar_label: "Kernel traces/metrics (eBPF)"
6
-learn_status: "Published"
7
-learn_topic_type: "References"
8
-learn_rel_path: "Integrations/Monitor/System metrics"
9
--->
10
-
1
# Kernel traces/metrics (eBPF) collector
2
13
-The Netdata Agent provides many [eBPF](https://ebpf.io/what-is-ebpf/) programs to help you troubleshoot and debug how applications interact with the Linux kernel. The `ebpf.plugin` uses [tracepoints, trampoline, and2 kprobes](#how-netdata-collects-data-using-probes-and-tracepoints) to collect a wide array of high value data about the host that would otherwise be impossible to capture.
3
+The Netdata Agent provides many [eBPF](https://ebpf.io/what-is-ebpf/) programs to help you troubleshoot and debug how applications interact with the Linux kernel. The `ebpf.plugin` uses [tracepoints, trampoline, and2 kprobes](#how-netdata-collects-data-using-probes-and-tracepoints) to collect a wide array of high value data about the host that would otherwise be impossible to capture.
4
5
> ❗ eBPF monitoring only works on Linux systems and with specific Linux kernels, including all kernels newer than `4.11.0`, and all kernels on CentOS 7.6 or later. For kernels older than `4.11.0`, improved support is in active development.
6
@@ -26,10 +16,10 @@ For hands-on configuration and troubleshooting tips see our [tutorial on trouble
16
17
Netdata uses the following features from the Linux kernel to run eBPF programs:
18
29
-- Tracepoints are hooks to call specific functions. Tracepoints are more stable than `kprobes` and are preferred when
19
+- Tracepoints are hooks to call specific functions. Tracepoints are more stable than `kprobes` and are preferred when
20
both options are available.
31
-- Trampolines are bridges between kernel functions, and BPF programs. Netdata uses them by default whenever available.
32
-- Kprobes and return probes (`kretprobe`): Probes can insert virtually into any kernel instruction. When eBPF runs in `entry` mode, it attaches only `kprobes` for internal functions monitoring calls and some arguments every time a function is called. The user can also change configuration to use [`return`](#global-configuration-options) mode, and this will allow users to monitor return from these functions and detect possible failures.
21
+- Trampolines are bridges between kernel functions, and BPF programs. Netdata uses them by default whenever available.
22
+- Kprobes and return probes (`kretprobe`): Probes can insert virtually into any kernel instruction. When eBPF runs in `entry` mode, it attaches only `kprobes` for internal functions monitoring calls and some arguments every time a function is called. The user can also change configuration to use [`return`](#global-configuration-options) mode, and this will allow users to monitor return from these functions and detect possible failures.
23
24
In each case, wherever a normal kprobe, kretprobe, or tracepoint would have run its hook function, an eBPF program is run instead, performing various collection logic before letting the kernel continue its normal control flow.
25
@@ -38,24 +28,25 @@ There are more methods to trigger eBPF programs, such as uprobes, but currently
28
## Configuring ebpf.plugin
29
30
The eBPF collector is installed and enabled by default on most new installations of the Agent.
41
-If your Agent is v1.22 or older, you may to enable the collector yourself.
31
+If your Agent is v1.22 or older, you may to enable the collector yourself.
32
33
### Enable the eBPF collector
34
45
-To enable or disable the entire eBPF collector:
35
+To enable or disable the entire eBPF collector:
36
+
37
+1. Navigate to the [Netdata config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory).
38
47
-1. Navigate to the [Netdata config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory).
39
```bash
40
cd /etc/netdata
41
```
42
52
-2. Use the [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) script to edit `netdata.conf`.
43
+2. Use the [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) script to edit `netdata.conf`.
44
45
```bash
46
./edit-config netdata.conf
47
```
48
58
-3. Enable the collector by scrolling down to the `[plugins]` section. Uncomment the line `ebpf` (not
49
+3. Enable the collector by scrolling down to the `[plugins]` section. Uncomment the line `ebpf` (not
50
`ebpf_process`) and set it to `yes`.
51
52
```conf
@@ -65,15 +56,17 @@ To enable or disable the entire eBPF collector:
56
57
### Configure the eBPF collector
58
68
-You can configure the eBPF collector's behavior to fine-tune which metrics you receive and [optimize performance]\(#performance opimization).
59
+You can configure the eBPF collector's behavior to fine-tune which metrics you receive and [optimize performance](#performance-opimization).
60
61
To edit the `ebpf.d.conf`:
62
72
-1. Navigate to the [Netdata config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory).
63
+1. Navigate to the [Netdata config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory).
64
+
65
```bash
66
cd /etc/netdata
67
```
76
-2. Use the [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) script to edit [`ebpf.d.conf`](https://github.com/netdata/netdata/blob/master/src/collectors/ebpf.plugin/ebpf.d.conf).
68
+
69
+2. Use the [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) script to edit [`ebpf.d.conf`](https://github.com/netdata/netdata/blob/master/src/collectors/ebpf.plugin/ebpf.d.conf).
70
71
```bash
72
./edit-config ebpf.d.conf
@@ -94,9 +87,9 @@ By default, this plugin uses the `entry` mode. Changing this mode can create sig
87
system, but also offer valuable information if you are developing or debugging software. The `ebpf load mode` option
88
accepts the following values:
89
97
-- `entry`: This is the default mode. In this mode, the eBPF collector only monitors calls for the functions described in
90
+- `entry`: This is the default mode. In this mode, the eBPF collector only monitors calls for the functions described in
91
the sections above, and does not show charts related to errors.
99
-- `return`: In the `return` mode, the eBPF collector monitors the same kernel functions as `entry`, but also creates new
92
+- `return`: In the `return` mode, the eBPF collector monitors the same kernel functions as `entry`, but also creates new
93
charts for the return of these functions, such as errors. Monitoring function returns can help in debugging software,
94
such as failing to close file descriptors or creating zombie processes.
95
@@ -133,10 +126,7 @@ If you do not need to monitor specific metrics for your `cgroups`, you can enabl
126
127
#### Maps per Core
128
136
-When netdata is running on kernels newer than `4.6` users are allowed to modify how the `ebpf.plugin` creates maps (hash or
137
-array). When `maps per core` is defined as `yes`, plugin will create a map per core on host, on the other hand,
138
-when the value is set as `no` only one hash table will be created, this option will use less memory, but it also can
139
-increase overhead for processes.
129
+When netdata is running on kernels newer than `4.6` users are allowed to modify how the `ebpf.plugin` creates maps (hash or array). When `maps per core` is defined as `yes`, plugin will create a map per core on host, on the other hand, when the value is set as `no` only one hash table will be created, this option will use less memory, but it also can increase overhead for processes.
130
131
#### Collect PID
132
@@ -146,10 +136,10 @@ process group for which it needs to plot data.
136
There are different ways to collect PID, and you can select the way `ebpf.plugin` collects data with the following
137
values:
138
149
-- `real parent`: This is the default mode. Collection will aggregate data for the real parent, the thread that creates
139
+- `real parent`: This is the default mode. Collection will aggregate data for the real parent, the thread that creates
140
child threads.
151
-- `parent`: Parent and real parent are the same when a process starts, but this value can be changed during run time.
152
-- `all`: This option will store all PIDs that run on the host. Note, this method can be expensive for the host,
141
+- `parent`: Parent and real parent are the same when a process starts, but this value can be changed during run time.
142
+- `all`: This option will store all PIDs that run on the host. Note, this method can be expensive for the host,
143
because more memory needs to be allocated and parsed.
144
145
The threads that have integration with other collectors have an internal clean up wherein they attach either a
@@ -174,97 +164,97 @@ Linux metrics:
164
165
> Note: The parenthetical accompanying each bulleted item provides the chart name.
166
177
-- mem
178
- - Number of processes killed due out of memory. (`oomkills`)
179
-- process
180
- - Number of processes created with `do_fork`. (`process_create`)
181
- - Number of threads created with `do_fork` or `clone (2)`, depending on your system's kernel
167
+- mem
168
+ - Number of processes killed due out of memory. (`oomkills`)
169
+- process
170
+ - Number of processes created with `do_fork`. (`process_create`)
171
+ - Number of threads created with `do_fork` or `clone (2)`, depending on your system's kernel
172
version. (`thread_create`)
183
- - Number of times that a process called `do_exit`. (`task_exit`)
184
- - Number of times that a process called `release_task`. (`task_close`)
185
- - Number of times that an error happened to create thread or process. (`task_error`)
186
-- swap
187
- - Number of calls to `swap_readpage`. (`swap_read_call`)
188
- - Number of calls to `swap_writepage`. (`swap_write_call`)
189
-- network
190
- - Number of outbound connections using TCP/IPv4. (`outbound_conn_ipv4`)
191
- - Number of outbound connections using TCP/IPv6. (`outbound_conn_ipv6`)
192
- - Number of bytes sent. (`total_bandwidth_sent`)
193
- - Number of bytes received. (`total_bandwidth_recv`)
194
- - Number of calls to `tcp_sendmsg`. (`bandwidth_tcp_send`)
195
- - Number of calls to `tcp_cleanup_rbuf`. (`bandwidth_tcp_recv`)
196
- - Number of calls to `tcp_retransmit_skb`. (`bandwidth_tcp_retransmit`)
197
- - Number of calls to `udp_sendmsg`. (`bandwidth_udp_send`)
198
- - Number of calls to `udp_recvmsg`. (`bandwidth_udp_recv`)
199
-- file access
200
- - Number of calls to open files. (`file_open`)
201
- - Number of calls to open files that returned errors. (`open_error`)
202
- - Number of files closed. (`file_closed`)
203
- - Number of calls to close files that returned errors. (`file_error_closed`)
204
-- vfs
205
- - Number of calls to `vfs_unlink`. (`file_deleted`)
206
- - Number of calls to `vfs_write`. (`vfs_write_call`)
207
- - Number of calls to write a file that returned errors. (`vfs_write_error`)
208
- - Number of calls to `vfs_read`. (`vfs_read_call`)
209
- - - Number of calls to read a file that returned errors. (`vfs_read_error`)
210
- - Number of bytes written with `vfs_write`. (`vfs_write_bytes`)
211
- - Number of bytes read with `vfs_read`. (`vfs_read_bytes`)
212
- - Number of calls to `vfs_fsync`. (`vfs_fsync`)
213
- - Number of calls to sync file that returned errors. (`vfs_fsync_error`)
214
- - Number of calls to `vfs_open`. (`vfs_open`)
215
- - Number of calls to open file that returned errors. (`vfs_open_error`)
216
- - Number of calls to `vfs_create`. (`vfs_create`)
217
- - Number of calls to open file that returned errors. (`vfs_create_error`)
218
-- page cache
219
- - Ratio of pages accessed. (`cachestat_ratio`)
220
- - Number of modified pages ("dirty"). (`cachestat_dirties`)
221
- - Number of accessed pages. (`cachestat_hits`)
222
- - Number of pages brought from disk. (`cachestat_misses`)
223
-- directory cache
224
- - Ratio of files available in directory cache. (`dc_hit_ratio`)
225
- - Number of files accessed. (`dc_reference`)
226
- - Number of files accessed that were not in cache. (`dc_not_cache`)
227
- - Number of files not found. (`dc_not_found`)
228
-- ipc shm
229
- - Number of calls to `shm_get`. (`shmget_call`)
230
- - Number of calls to `shm_at`. (`shmat_call`)
231
- - Number of calls to `shm_dt`. (`shmdt_call`)
232
- - Number of calls to `shm_ctl`. (`shmctl_call`)
173
+ - Number of times that a process called `do_exit`. (`task_exit`)
174
+ - Number of times that a process called `release_task`. (`task_close`)
175
+ - Number of times that an error happened to create thread or process. (`task_error`)
176
+- swap
177
+ - Number of calls to `swap_readpage`. (`swap_read_call`)
178
+ - Number of calls to `swap_writepage`. (`swap_write_call`)
179
+- network
180
+ - Number of outbound connections using TCP/IPv4. (`outbound_conn_ipv4`)
181
+ - Number of outbound connections using TCP/IPv6. (`outbound_conn_ipv6`)
182
+ - Number of bytes sent. (`total_bandwidth_sent`)
183
+ - Number of bytes received. (`total_bandwidth_recv`)
184
+ - Number of calls to `tcp_sendmsg`. (`bandwidth_tcp_send`)
185
+ - Number of calls to `tcp_cleanup_rbuf`. (`bandwidth_tcp_recv`)
186
+ - Number of calls to `tcp_retransmit_skb`. (`bandwidth_tcp_retransmit`)
187
+ - Number of calls to `udp_sendmsg`. (`bandwidth_udp_send`)
188
+ - Number of calls to `udp_recvmsg`. (`bandwidth_udp_recv`)
189
+- file access
190
+ - Number of calls to open files. (`file_open`)
191
+ - Number of calls to open files that returned errors. (`open_error`)
192
+ - Number of files closed. (`file_closed`)
193
+ - Number of calls to close files that returned errors. (`file_error_closed`)
194
+- vfs
195
+ - Number of calls to `vfs_unlink`. (`file_deleted`)
196
+ - Number of calls to `vfs_write`. (`vfs_write_call`)
197
+ - Number of calls to write a file that returned errors. (`vfs_write_error`)
198
+ - Number of calls to `vfs_read`. (`vfs_read_call`)
199
+ - - Number of calls to read a file that returned errors. (`vfs_read_error`)
200
+ - Number of bytes written with `vfs_write`. (`vfs_write_bytes`)
201
+ - Number of bytes read with `vfs_read`. (`vfs_read_bytes`)
202
+ - Number of calls to `vfs_fsync`. (`vfs_fsync`)
203
+ - Number of calls to sync file that returned errors. (`vfs_fsync_error`)
204
+ - Number of calls to `vfs_open`. (`vfs_open`)
205
+ - Number of calls to open file that returned errors. (`vfs_open_error`)
206
+ - Number of calls to `vfs_create`. (`vfs_create`)
207
+ - Number of calls to open file that returned errors. (`vfs_create_error`)
208
+- page cache
209
+ - Ratio of pages accessed. (`cachestat_ratio`)
210
+ - Number of modified pages ("dirty"). (`cachestat_dirties`)
211
+ - Number of accessed pages. (`cachestat_hits`)
212
+ - Number of pages brought from disk. (`cachestat_misses`)
213
+- directory cache
214
+ - Ratio of files available in directory cache. (`dc_hit_ratio`)
215
+ - Number of files accessed. (`dc_reference`)
216
+ - Number of files accessed that were not in cache. (`dc_not_cache`)
217
+ - Number of files not found. (`dc_not_found`)
218
+- ipc shm
219
+ - Number of calls to `shm_get`. (`shmget_call`)
220
+ - Number of calls to `shm_at`. (`shmat_call`)
221
+ - Number of calls to `shm_dt`. (`shmdt_call`)
222
+ - Number of calls to `shm_ctl`. (`shmctl_call`)
223
224
### `[ebpf programs]` configuration options
225
226
The eBPF collector enables and runs the following eBPF programs by default:
227
238
-- `cachestat`: Netdata's eBPF data collector creates charts about the memory page cache. When the integration with
228
+- `cachestat`: Netdata's eBPF data collector creates charts about the memory page cache. When the integration with
229
[`apps.plugin`](/src/collectors/apps.plugin/README.md) is enabled, this collector creates charts for the whole host _and_
230
for each application.
241
-- `fd` : This eBPF program creates charts that show information about calls to open files.
242
-- `mount`: This eBPF program creates charts that show calls to syscalls mount(2) and umount(2).
243
-- `shm`: This eBPF program creates charts that show calls to syscalls shmget(2), shmat(2), shmdt(2) and shmctl(2).
244
-- `process`: This eBPF program creates charts that show information about process life. When in `return` mode, it also
231
+- `fd` : This eBPF program creates charts that show information about calls to open files.
232
+- `mount`: This eBPF program creates charts that show calls to syscalls mount(2) and umount(2).
233
+- `shm`: This eBPF program creates charts that show calls to syscalls shmget(2), shmat(2), shmdt(2) and shmctl(2).
234
+- `process`: This eBPF program creates charts that show information about process life. When in `return` mode, it also
235
creates charts showing errors when these operations are executed.
246
-- `hardirq`: This eBPF program creates charts that show information about time spent servicing individual hardware
236
+- `hardirq`: This eBPF program creates charts that show information about time spent servicing individual hardware
237
interrupt requests (hard IRQs).
248
-- `softirq`: This eBPF program creates charts that show information about time spent servicing individual software
238
+- `softirq`: This eBPF program creates charts that show information about time spent servicing individual software
239
interrupt requests (soft IRQs).
250
-- `oomkill`: This eBPF program creates a chart that shows OOM kills for all applications recognized via
240
+- `oomkill`: This eBPF program creates a chart that shows OOM kills for all applications recognized via
241
the `apps.plugin` integration. Note that this program will show application charts regardless of whether apps
242
integration is turned on or off.
243
244
You can also enable the following eBPF programs:
245
256
-- `dcstat` : This eBPF program creates charts that show information about file access using directory cache. It appends
246
+- `dcstat` : This eBPF program creates charts that show information about file access using directory cache. It appends
247
`kprobes` for `lookup_fast()` and `d_lookup()` to identify if files are inside directory cache, outside and files are
248
not found.
259
-- `disk` : This eBPF program creates charts that show information about disk latency independent of filesystem.
260
-- `filesystem` : This eBPF program creates charts that show information about some filesystem latency.
261
-- `swap` : This eBPF program creates charts that show information about swap access.
262
-- `mdflush`: This eBPF program creates charts that show information about
263
-- `sync`: Monitor calls to syscalls sync(2), fsync(2), fdatasync(2), syncfs(2), msync(2), and sync_file_range(2).
264
-- `socket`: This eBPF program creates charts with information about `TCP` and `UDP` functions, including the
249
+- `disk` : This eBPF program creates charts that show information about disk latency independent of filesystem.
250
+- `filesystem` : This eBPF program creates charts that show information about some filesystem latency.
251
+- `swap` : This eBPF program creates charts that show information about swap access.
252
+- `mdflush`: This eBPF program creates charts that show information about
253
+- `sync`: Monitor calls to syscalls sync(2), fsync(2), fdatasync(2), syncfs(2), msync(2), and sync_file_range(2).
254
+- `socket`: This eBPF program creates charts with information about `TCP` and `UDP` functions, including the
255
bandwidth consumed by each.
256
multi-device software flushes.
267
-- `vfs`: This eBPF program creates charts that show information about VFS (Virtual File System) functions.
257
+- `vfs`: This eBPF program creates charts that show information about VFS (Virtual File System) functions.
258
259
### Configuring eBPF threads
260
@@ -272,24 +262,26 @@ You can configure each thread of the eBPF data collector. This allows you to ove
262
263
To configure an eBPF thread:
264
275
-1. Navigate to the [Netdata config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory).
265
+1. Navigate to the [Netdata config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory).
266
+
267
```bash
268
cd /etc/netdata
269
```
279
-2. Use the [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) script to edit a thread configuration file. The following configuration files are available:
270
281
- - `network.conf`: Configuration for the [`network` thread](#network-configuration). This config file overwrites the global options and also
271
+2. Use the [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) script to edit a thread configuration file. The following configuration files are available:
272
+
273
+ - `network.conf`: Configuration for the [`network` thread](#network-configuration). This config file overwrites the global options and also
274
lets you specify which network the eBPF collector monitors.
283
- - `process.conf`: Configuration for the [`process` thread](#sync-configuration).
284
- - `cachestat.conf`: Configuration for the `cachestat` thread(#filesystem-configuration).
285
- - `dcstat.conf`: Configuration for the `dcstat` thread.
286
- - `disk.conf`: Configuration for the `disk` thread.
287
- - `fd.conf`: Configuration for the `file descriptor` thread.
288
- - `filesystem.conf`: Configuration for the `filesystem` thread.
289
- - `hardirq.conf`: Configuration for the `hardirq` thread.
290
- - `softirq.conf`: Configuration for the `softirq` thread.
291
- - `sync.conf`: Configuration for the `sync` thread.
292
- - `vfs.conf`: Configuration for the `vfs` thread.
275
+ - `process.conf`: Configuration for the [`process` thread](#sync-configuration).
276
+ - `cachestat.conf`: Configuration for the `cachestat` thread(#filesystem-configuration).
277
+ - `dcstat.conf`: Configuration for the `dcstat` thread.
278
+ - `disk.conf`: Configuration for the `disk` thread.
279
+ - `fd.conf`: Configuration for the `file descriptor` thread.
280
+ - `filesystem.conf`: Configuration for the `filesystem` thread.
281
+ - `hardirq.conf`: Configuration for the `hardirq` thread.
282
+ - `softirq.conf`: Configuration for the `softirq` thread.
283
+ - `sync.conf`: Configuration for the `sync` thread.
284
+ - `vfs.conf`: Configuration for the `vfs` thread.
285
286
```bash
287
./edit-config FILE.conf
@@ -324,13 +316,13 @@ and `145`.
316
317
The following options are available:
318
327
-- `enabled`: Disable network connections monitoring. This can affect directly some funcion output.
328
-- `resolve hostname ips`: Enable resolving IPs to hostnames. It is disabled by default because it can be too slow.
329
-- `resolve service names`: Convert destination ports into service names, for example, port `53` protocol `UDP` becomes `domain`.
319
+- `enabled`: Disable network connections monitoring. This can affect directly some funcion output.
320
+- `resolve hostname ips`: Enable resolving IPs to hostnames. It is disabled by default because it can be too slow.
321
+- `resolve service names`: Convert destination ports into service names, for example, port `53` protocol `UDP` becomes `domain`.
322
all names are read from /etc/services.
331
-- `ports`: Define the destination ports for Netdata to monitor.
332
-- `hostnames`: The list of hostnames that can be resolved to an IP address.
333
-- `ips`: The IP or range of IPs that you want to monitor. You can use IPv4 or IPv6 addresses, use dashes to define a
323
+- `ports`: Define the destination ports for Netdata to monitor.
324
+- `hostnames`: The list of hostnames that can be resolved to an IP address.
325
+- `ips`: The IP or range of IPs that you want to monitor. You can use IPv4 or IPv6 addresses, use dashes to define a
326
range of IPs, or use CIDR values.
327
328
By default the traffic table is created using the destination IPs and ports of the sockets. This can be
@@ -408,19 +400,18 @@ You can run our helper script to determine whether your system can support eBPF
400
curl -sSL https://raw.githubusercontent.com/netdata/kernel-collector/master/tools/check-kernel-config.sh | sudo bash
401
```
402
411
-
403
If you see a warning about a missing kernel
404
configuration (`KPROBES KPROBES_ON_FTRACE HAVE_KPROBES BPF BPF_SYSCALL BPF_JIT`), you will need to recompile your kernel
405
to support this configuration. The process of recompiling Linux kernels varies based on your distribution and version.
406
Read the documentation for your system's distribution to learn more about the specific workflow for recompiling the
407
kernel, ensuring that you set all the necessary
408
418
-- [Ubuntu](https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel)
419
-- [Debian](https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-common-official)
420
-- [Fedora](https://fedoraproject.org/wiki/Building_a_custom_kernel)
421
-- [CentOS](https://wiki.centos.org/HowTos/Custom_Kernel)
422
-- [Arch Linux](https://wiki.archlinux.org/index.php/Kernel/Traditional_compilation)
423
-- [Slackware](https://docs.slackware.com/howtos:slackware_admin:kernelbuilding)
409
+- [Ubuntu](https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel)
410
+- [Debian](https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-common-official)
411
+- [Fedora](https://fedoraproject.org/wiki/Building_a_custom_kernel)
412
+- [CentOS](https://wiki.centos.org/HowTos/Custom_Kernel)
413
+- [Arch Linux](https://wiki.archlinux.org/index.php/Kernel/Traditional_compilation)
414
+- [Slackware](https://docs.slackware.com/howtos:slackware_admin:kernelbuilding)
415
416
### Mount `debugfs` and `tracefs`
417
@@ -455,12 +446,12 @@ Internally, the Linux kernel treats both processes and threads as `tasks`. To cr
446
system calls: `fork(2)`, `vfork(2)`, and `clone(2)`. To generate this chart, the eBPF
447
collector uses the following `tracepoints` and `kprobe`:
448
458
-- `sched/sched_process_fork`: Tracepoint called after a call for `fork (2)`, `vfork (2)` and `clone (2)`.
459
-- `sched/sched_process_exec`: Tracepoint called after a exec-family syscall.
460
-- `kprobe/kernel_clone`: This is the main [`fork()`](https://elixir.bootlin.com/linux/v5.10/source/kernel/fork.c#L2415)
449
+- `sched/sched_process_fork`: Tracepoint called after a call for `fork (2)`, `vfork (2)` and `clone (2)`.
450
+- `sched/sched_process_exec`: Tracepoint called after a exec-family syscall.
451
+- `kprobe/kernel_clone`: This is the main [`fork()`](https://elixir.bootlin.com/linux/v5.10/source/kernel/fork.c#L2415)
452
routine since kernel `5.10.0` was released.
462
-- `kprobe/_do_fork`: Like `kernel_clone`, but this was the main function between kernels `4.2.0` and `5.9.16`
463
-- `kprobe/do_fork`: This was the main function before kernel `4.2.0`.
453
+- `kprobe/_do_fork`: Like `kernel_clone`, but this was the main function between kernels `4.2.0` and `5.9.16`
454
+- `kprobe/do_fork`: This was the main function before kernel `4.2.0`.
455
456
#### Process Exit
457
@@ -469,8 +460,8 @@ system that the task is finishing its work. The second step is to release the ke
460
function `release_task`. The difference between the two dimensions can help you discover
461
[zombie processes](https://en.wikipedia.org/wiki/Zombie_process). To get the metrics, the collector uses:
462
472
-- `sched/sched_process_exit`: Tracepoint called after a task exits.
473
-- `kprobe/release_task`: This function is called when a process exits, as the kernel still needs to remove the process
463
+- `sched/sched_process_exit`: Tracepoint called after a task exits.
464
+- `kprobe/release_task`: This function is called when a process exits, as the kernel still needs to remove the process
465
descriptor.
466
467
#### Task error
@@ -489,9 +480,9 @@ the collector attaches `kprobes` for cited functions.
480
481
The following `tracepoints` are used to measure time usage for soft IRQs:
482
492
-- [`irq/softirq_entry`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_softirq_entry): Called
483
+- [`irq/softirq_entry`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_softirq_entry): Called
484
before softirq handler
494
-- [`irq/softirq_exit`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_softirq_exit): Called when
485
+- [`irq/softirq_exit`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_softirq_exit): Called when
486
softirq handler returns.
487
488
#### Hard IRQ
@@ -499,60 +490,60 @@ The following `tracepoints` are used to measure time usage for soft IRQs:
490
The following tracepoints are used to measure the latency of servicing a
491
hardware interrupt request (hard IRQ).
492
502
-- [`irq/irq_handler_entry`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_irq_handler_entry):
493
+- [`irq/irq_handler_entry`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_irq_handler_entry):
494
Called immediately before the IRQ action handler.
504
-- [`irq/irq_handler_exit`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_irq_handler_exit):
495
+- [`irq/irq_handler_exit`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_irq_handler_exit):
496
Called immediately after the IRQ action handler returns.
506
-- `irq_vectors`: These are traces from `irq_handler_entry` and
497
+- `irq_vectors`: These are traces from `irq_handler_entry` and
498
`irq_handler_exit` when an IRQ is handled. The following elements from vector
499
are triggered:
509
- - `irq_vectors/local_timer_entry`
510
- - `irq_vectors/local_timer_exit`
511
- - `irq_vectors/reschedule_entry`
512
- - `irq_vectors/reschedule_exit`
513
- - `irq_vectors/call_function_entry`
514
- - `irq_vectors/call_function_exit`
515
- - `irq_vectors/call_function_single_entry`
516
- - `irq_vectors/call_function_single_xit`
517
- - `irq_vectors/irq_work_entry`
518
- - `irq_vectors/irq_work_exit`
519
- - `irq_vectors/error_apic_entry`
520
- - `irq_vectors/error_apic_exit`
521
- - `irq_vectors/thermal_apic_entry`
522
- - `irq_vectors/thermal_apic_exit`
523
- - `irq_vectors/threshold_apic_entry`
524
- - `irq_vectors/threshold_apic_exit`
525
- - `irq_vectors/deferred_error_entry`
526
- - `irq_vectors/deferred_error_exit`
527
- - `irq_vectors/spurious_apic_entry`
528
- - `irq_vectors/spurious_apic_exit`
529
- - `irq_vectors/x86_platform_ipi_entry`
530
- - `irq_vectors/x86_platform_ipi_exit`
500
+ - `irq_vectors/local_timer_entry`
501
+ - `irq_vectors/local_timer_exit`
502
+ - `irq_vectors/reschedule_entry`
503
+ - `irq_vectors/reschedule_exit`
504
+ - `irq_vectors/call_function_entry`
505
+ - `irq_vectors/call_function_exit`
506
+ - `irq_vectors/call_function_single_entry`
507
+ - `irq_vectors/call_function_single_xit`
508
+ - `irq_vectors/irq_work_entry`
509
+ - `irq_vectors/irq_work_exit`
510
+ - `irq_vectors/error_apic_entry`
511
+ - `irq_vectors/error_apic_exit`
512
+ - `irq_vectors/thermal_apic_entry`
513
+ - `irq_vectors/thermal_apic_exit`
514
+ - `irq_vectors/threshold_apic_entry`
515
+ - `irq_vectors/threshold_apic_exit`
516
+ - `irq_vectors/deferred_error_entry`
517
+ - `irq_vectors/deferred_error_exit`
518
+ - `irq_vectors/spurious_apic_entry`
519
+ - `irq_vectors/spurious_apic_exit`
520
+ - `irq_vectors/x86_platform_ipi_entry`
521
+ - `irq_vectors/x86_platform_ipi_exit`
522
523
#### IPC shared memory
524
525
To monitor shared memory system call counts, Netdata attaches tracing in the following functions:
526
536
-- `shmget`: Runs when [`shmget`](https://man7.org/linux/man-pages/man2/shmget.2.html) is called.
537
-- `shmat`: Runs when [`shmat`](https://man7.org/linux/man-pages/man2/shmat.2.html) is called.
538
-- `shmdt`: Runs when [`shmdt`](https://man7.org/linux/man-pages/man2/shmat.2.html) is called.
539
-- `shmctl`: Runs when [`shmctl`](https://man7.org/linux/man-pages/man2/shmctl.2.html) is called.
527
+- `shmget`: Runs when [`shmget`](https://man7.org/linux/man-pages/man2/shmget.2.html) is called.
528
+- `shmat`: Runs when [`shmat`](https://man7.org/linux/man-pages/man2/shmat.2.html) is called.
529
+- `shmdt`: Runs when [`shmdt`](https://man7.org/linux/man-pages/man2/shmat.2.html) is called.
530
+- `shmctl`: Runs when [`shmctl`](https://man7.org/linux/man-pages/man2/shmctl.2.html) is called.
531
532
### Memory
533
534
In the memory submenu the eBPF plugin creates two submenus **page cache** and **synchronization** with the following
535
organization:
536
546
-- Page Cache
547
- - Page cache ratio
548
- - Dirty pages
549
- - Page cache hits
550
- - Page cache misses
551
-- Synchronization
552
- - File sync
553
- - Memory map sync
554
- - File system sync
555
- - File range sync
537
+- Page Cache
538
+ - Page cache ratio
539
+ - Dirty pages
540
+ - Page cache hits
541
+ - Page cache misses
542
+- Synchronization
543
+ - File sync
544
+ - Memory map sync
545
+ - File system sync
546
+ - File range sync
547
548
#### Page cache hits
549
@@ -587,10 +578,10 @@ The chart `cachestat_ratio` shows how processes are accessing page cache. In a n
578
100%, which means that the majority of the work on the machine is processed in memory. To calculate the ratio, Netdata
579
attaches `kprobes` for kernel functions:
580
590
-- `add_to_page_cache_lru`: Page addition.
591
-- `mark_page_accessed`: Access to cache.
592
-- `account_page_dirtied`: Dirty (modified) pages.
593
-- `mark_buffer_dirty`: Writes to page cache.
581
+- `add_to_page_cache_lru`: Page addition.
582
+- `mark_page_accessed`: Access to cache.
583
+- `account_page_dirtied`: Dirty (modified) pages.
584
+- `mark_buffer_dirty`: Writes to page cache.
585
586
#### Page cache misses
587
@@ -638,7 +629,7 @@ By default, MD flush is disabled. To enable it, configure your
629
630
To collect data related to Linux multi-device (MD) flushing, the following kprobe is used:
631
641
-- `kprobe/md_flush_request`: called whenever a request for flushing multi-device data is made.
632
+- `kprobe/md_flush_request`: called whenever a request for flushing multi-device data is made.
633
634
### Disk
635
@@ -648,9 +639,9 @@ The eBPF plugin also shows a chart in the Disk section when the `disk` thread is
639
640
This will create the chart `disk_latency_io` for each disk on the host. The following tracepoints are used:
641
651
-- [`block/block_rq_issue`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_block_rq_issue):
642
+- [`block/block_rq_issue`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_block_rq_issue):
643
IO request operation to a device drive.
653
-- [`block/block_rq_complete`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_block_rq_complete):
644
+- [`block/block_rq_complete`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_block_rq_complete):
645
IO operation completed by device.
646
647
Disk Latency is the single most important metric to focus on when it comes to storage performance, under most circumstances.
@@ -675,10 +666,10 @@ To measure the latency of executing some actions in an
666
collector needs to attach `kprobes` and `kretprobes` for each of the following
667
functions:
668
678
-- `ext4_file_read_iter`: Function used to measure read latency.
679
-- `ext4_file_write_iter`: Function used to measure write latency.
680
-- `ext4_file_open`: Function used to measure open latency.
681
-- `ext4_sync_file`: Function used to measure sync latency.
669
+- `ext4_file_read_iter`: Function used to measure read latency.
670
+- `ext4_file_write_iter`: Function used to measure write latency.
671
+- `ext4_file_open`: Function used to measure open latency.
672
+- `ext4_sync_file`: Function used to measure sync latency.
673
674
#### ZFS
675
@@ -686,10 +677,10 @@ To measure the latency of executing some actions in a zfs filesystem, the
677
collector needs to attach `kprobes` and `kretprobes` for each of the following
678
functions:
679
689
-- `zpl_iter_read`: Function used to measure read latency.
690
-- `zpl_iter_write`: Function used to measure write latency.
691
-- `zpl_open`: Function used to measure open latency.
692
-- `zpl_fsync`: Function used to measure sync latency.
680
+- `zpl_iter_read`: Function used to measure read latency.
681
+- `zpl_iter_write`: Function used to measure write latency.
682
+- `zpl_open`: Function used to measure open latency.
683
+- `zpl_fsync`: Function used to measure sync latency.
684
685
#### XFS
686
@@ -698,10 +689,10 @@ To measure the latency of executing some actions in an
689
collector needs to attach `kprobes` and `kretprobes` for each of the following
690
functions:
691
701
-- `xfs_file_read_iter`: Function used to measure read latency.
702
-- `xfs_file_write_iter`: Function used to measure write latency.
703
-- `xfs_file_open`: Function used to measure open latency.
704
-- `xfs_file_fsync`: Function used to measure sync latency.
692
+- `xfs_file_read_iter`: Function used to measure read latency.
693
+- `xfs_file_write_iter`: Function used to measure write latency.
694
+- `xfs_file_open`: Function used to measure open latency.
695
+- `xfs_file_fsync`: Function used to measure sync latency.
696
697
#### NFS
698
@@ -710,11 +701,11 @@ To measure the latency of executing some actions in an
701
collector needs to attach `kprobes` and `kretprobes` for each of the following
702
functions:
703
713
-- `nfs_file_read`: Function used to measure read latency.
714
-- `nfs_file_write`: Function used to measure write latency.
715
-- `nfs_file_open`: Functions used to measure open latency.
716
-- `nfs4_file_open`: Functions used to measure open latency for NFS v4.
717
-- `nfs_getattr`: Function used to measure sync latency.
704
+- `nfs_file_read`: Function used to measure read latency.
705
+- `nfs_file_write`: Function used to measure write latency.
706
+- `nfs_file_open`: Functions used to measure open latency.
707
+- `nfs4_file_open`: Functions used to measure open latency for NFS v4.
708
+- `nfs_getattr`: Function used to measure sync latency.
709
710
#### btrfs
711
@@ -724,24 +715,24 @@ filesystem, the collector needs to attach `kprobes` and `kretprobes` for each of
715
> Note: We are listing two functions used to measure `read` latency, but we use either `btrfs_file_read_iter` or
716
> `generic_file_read_iter`, depending on kernel version.
717
727
-- `btrfs_file_read_iter`: Function used to measure read latency since kernel `5.10.0`.
728
-- `generic_file_read_iter`: Like `btrfs_file_read_iter`, but this function was used before kernel `5.10.0`.
729
-- `btrfs_file_write_iter`: Function used to write data.
730
-- `btrfs_file_open`: Function used to open files.
731
-- `btrfs_sync_file`: Function used to synchronize data to filesystem.
718
+- `btrfs_file_read_iter`: Function used to measure read latency since kernel `5.10.0`.
719
+- `generic_file_read_iter`: Like `btrfs_file_read_iter`, but this function was used before kernel `5.10.0`.
720
+- `btrfs_file_write_iter`: Function used to write data.
721
+- `btrfs_file_open`: Function used to open files.
722
+- `btrfs_sync_file`: Function used to synchronize data to filesystem.
723
724
#### File descriptor
725
726
To give metrics related to `open` and `close` events, instead of attaching kprobes for each syscall used to do these
727
events, the collector attaches `kprobes` for the common function used for syscalls:
728
738
-- [`do_sys_open`](https://0xax.gitbooks.io/linux-insides/content/SysCall/linux-syscall-5.html): Internal function used to
729
+- [`do_sys_open`](https://0xax.gitbooks.io/linux-insides/content/SysCall/linux-syscall-5.html): Internal function used to
730
open files.
740
-- [`do_sys_openat2`](https://elixir.bootlin.com/linux/v5.6/source/fs/open.c#L1162):
731
+- [`do_sys_openat2`](https://elixir.bootlin.com/linux/v5.6/source/fs/open.c#L1162):
732
Function called from `do_sys_open` since version `5.6.0`.
742
-- [`close_fd`](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg2271761.html): Function used to close file
733
+- [`close_fd`](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg2271761.html): Function used to close file
734
descriptor since kernel `5.11.0`.
744
-- `__close_fd`: Function used to close files before version `5.11.0`.
735
+- `__close_fd`: Function used to close files before version `5.11.0`.
736
737
#### File error
738
@@ -761,21 +752,21 @@ To measure the latency and total quantity of executing some VFS-level
752
functions, ebpf.plugin needs to attach kprobes and kretprobes for each of the
753
following functions:
754
764
-- `vfs_write`: Function used monitoring the number of successful & failed
755
+- `vfs_write`: Function used monitoring the number of successful & failed
756
filesystem write calls, as well as the total number of written bytes.
766
-- `vfs_writev`: Same function as `vfs_write` but for vector writes (i.e. a
757
+- `vfs_writev`: Same function as `vfs_write` but for vector writes (i.e. a
758
single write operation using a group of buffers rather than 1).
768
-- `vfs_read`: Function used for monitoring the number of successful & failed
759
+- `vfs_read`: Function used for monitoring the number of successful & failed
760
filesystem read calls, as well as the total number of read bytes.
770
-- `vfs_readv` Same function as `vfs_read` but for vector reads (i.e. a single
761
+- `vfs_readv` Same function as `vfs_read` but for vector reads (i.e. a single
762
read operation using a group of buffers rather than 1).
772
-- `vfs_unlink`: Function used for monitoring the number of successful & failed
763
+- `vfs_unlink`: Function used for monitoring the number of successful & failed
764
filesystem unlink calls.
774
-- `vfs_fsync`: Function used for monitoring the number of successful & failed
765
+- `vfs_fsync`: Function used for monitoring the number of successful & failed
766
filesystem fsync calls.
776
-- `vfs_open`: Function used for monitoring the number of successful & failed
767
+- `vfs_open`: Function used for monitoring the number of successful & failed
768
filesystem open calls.
778
-- `vfs_create`: Function used for monitoring the number of successful & failed
769
+- `vfs_create`: Function used for monitoring the number of successful & failed
770
filesystem create calls.
771
772
##### VFS Deleted objects
@@ -816,8 +807,8 @@ Metrics for directory cache are collected using kprobe for `lookup_fast`, becaus
807
times this function is accessed. On the other hand, for `d_lookup` we are not only interested in the number of times it
808
is accessed, but also in possible errors, so we need to attach a `kretprobe`. For this reason, the following is used:
809
819
-- [`lookup_fast`](https://lwn.net/Articles/649115/): Called to look at data inside the directory cache.
820
-- [`d_lookup`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/dcache.c?id=052b398a43a7de8c68c13e7fa05d6b3d16ce6801#n2223):
810
+- [`lookup_fast`](https://lwn.net/Articles/649115/): Called to look at data inside the directory cache.
811
+- [`d_lookup`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/dcache.c?id=052b398a43a7de8c68c13e7fa05d6b3d16ce6801#n2223):
812
Called when the desired file is not inside the directory cache.
813
814
##### Directory Cache Interpretation
@@ -830,8 +821,8 @@ accessed before.
821
822
The following `tracing` are used to collect `mount` & `unmount` call counts:
823
833
-- [`mount`](https://man7.org/linux/man-pages/man2/mount.2.html): mount filesystem on host.
834
-- [`umount`](https://man7.org/linux/man-pages/man2/umount.2.html): umount filesystem on host.
824
+- [`mount`](https://man7.org/linux/man-pages/man2/mount.2.html): mount filesystem on host.
825
+- [`umount`](https://man7.org/linux/man-pages/man2/umount.2.html): umount filesystem on host.
826
827
### Networking Stack
828
@@ -855,10 +846,10 @@ to send & receive data and to close connections when `TCP` protocol is used.
846
847
This chart demonstrates calls to functions:
848
858
-- `tcp_sendmsg`: Function responsible to send data for a specified destination.
859
-- `tcp_cleanup_rbuf`: We use this function instead of `tcp_recvmsg`, because the last one misses `tcp_read_sock` traffic
849
+- `tcp_sendmsg`: Function responsible to send data for a specified destination.
850
+- `tcp_cleanup_rbuf`: We use this function instead of `tcp_recvmsg`, because the last one misses `tcp_read_sock` traffic
851
and we would also need to add more `tracing` to get the socket and package size.
861
-- `tcp_close`: Function responsible to close connection.
852
+- `tcp_close`: Function responsible to close connection.
853
854
#### TCP retransmit
855
@@ -881,7 +872,7 @@ calls, it monitors the number of bytes sent and received.
872
873
These are tracepoints related to [OOM](https://en.wikipedia.org/wiki/Out_of_memory) killing processes.
874
884
-- `oom/mark_victim`: Monitors when an oomkill event happens.
875
+- `oom/mark_victim`: Monitors when an oomkill event happens.
876
877
## Known issues
878
@@ -897,15 +888,14 @@ node is experiencing high memory usage and there is no obvious culprit to be fou
888
- Disable [integration with apps](#integration-with-appsplugin).
889
- Disable [integration with cgroup](#integration-with-cgroupsplugin).
890
900
-If with these changes you still suspect eBPF using too much memory, and there is no obvious culprit to be found
891
+If with these changes you still suspect eBPF using too much memory, and there is no obvious culprit to be found
892
in the `apps.mem` chart, consider testing for high kernel memory usage by [disabling eBPF monitoring](#configuring-ebpfplugin).
902
-Next, [restart Netdata](/packaging/installer/README.md#maintaining-a-netdata-agent-installation) with
903
-`sudo systemctl restart netdata` to see if system memory usage (see the `system.ram` chart) has dropped significantly.
893
+Next, [restart Netdata](/docs/netdata-agent/start-stop-restart.md) to see if system memory usage (see the `system.ram` chart) has dropped significantly.
894
895
Beginning with `v1.31`, kernel memory usage is configurable via the [`pid table size` setting](#pid-table-size)
896
in `ebpf.conf`.
897
908
-The total memory usage is a well known [issue](https://lore.kernel.org/all/167821082315.1693.6957546778534183486.git-patchwork-notify@kernel.org/)
898
+The total memory usage is a well known [issue](https://lore.kernel.org/all/167821082315.1693.6957546778534183486.git-patchwork-notify@kernel.org/)
899
for eBPF, this is not a bug present in plugin.
900
901
### SELinux
@@ -981,7 +971,7 @@ a feature called "lockdown," which may affect `ebpf.plugin` depending how the ke
971
shows how the lockdown module impacts `ebpf.plugin` based on the selected options:
972
973
| Enforcing kernel lockdown | Enable lockdown LSM early in init | Default lockdown mode | Can `ebpf.plugin` run with this? |
984
-| :------------------------ | :-------------------------------- | :-------------------- | :------------------------------- |
974
+|:--------------------------|:----------------------------------|:----------------------|:---------------------------------|
975
| YES | NO | NO | YES |
976
| YES | Yes | None | YES |
977
| YES | Yes | Integrity | YES |
src/collectors/freebsd.plugin/README.md
-11
@@ -1,16 +1,5 @@
1
-<!--
2
-title: "FreeBSD system metrics (freebsd.plugin)"
3
-custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/freebsd.plugin/README.md"
4
-sidebar_label: "FreeBSD system metrics (freebsd.plugin)"
5
-learn_status: "Published"
6
-learn_topic_type: "References"
7
-learn_rel_path: "Integrations/Monitor/System metrics"
8
--->
9
-
1
# FreeBSD system metrics (freebsd.plugin)
2
3
Collects resource usage and performance data on FreeBSD systems
4
5
By default, Netdata will enable monitoring metrics for disks, memory, and network only when they are not zero. If they are constantly zero they are ignored. Metrics that will start having values, after Netdata is started, will be detected and charts will be automatically added to the dashboard (a refresh of the dashboard is needed for them to appear though). Use `yes` instead of `auto` in plugin configuration sections to enable these charts permanently. You can also set the `enable zero metrics` option to `yes` in the `[global]` section which enables charts with zero metrics for all internal Netdata plugins.
15
-
16
-
src/collectors/log2journal/README.md
+17
-20
@@ -1,4 +1,3 @@
1
-
1
# log2journal
2
3
`log2journal` and `systemd-cat-native` can be used to convert a structured log file, such as the ones generated by web servers, into `systemd-journal` entries.
@@ -11,7 +10,6 @@ The result is like this: nginx logs into systemd-journal:
10
11
12
14
-
13
The overall process looks like this:
14
15
```bash
@@ -23,7 +21,8 @@ tail -F /var/log/nginx/*.log |\ # outputs log lines
21
These are the steps:
22
23
1. `tail -F /var/log/nginx/*.log`<br/>this command will tail all `*.log` files in `/var/log/nginx/`. We use `-F` instead of `-f` to ensure that files will still be tailed after log rotation.
26
-2. `log2joural` is a Netdata program. It reads log entries and extracts fields, according to the PCRE2 pattern it accepts. It can also apply some basic operations on the fields, like injecting new fields or duplicating existing ones or rewriting their values. The output of `log2journal` is in Systemd Journal Export Format, and it looks like this:
24
+2. `log2journal` is a Netdata program. It reads log entries and extracts fields, according to the PCRE2 pattern it accepts. It can also apply some basic operations on the fields, like injecting new fields or duplicating existing ones or rewriting their values. The output of `log2journal` is in Systemd Journal Export Format, and it looks like this:
25
+
26
```bash
27
KEY1=VALUE1 # << start of the first log line
28
KEY2=VALUE2
@@ -31,8 +30,8 @@ These are the steps:
30
KEY1=VALUE1 # << start of the second log line
31
KEY2=VALUE2
32
```
34
-3. `systemd-cat-native` is a Netdata program. I can send the logs to a local `systemd-journald` (journal namespaces supported), or to a remote `systemd-journal-remote`.
33
34
+3. `systemd-cat-native` is a Netdata program. I can send the logs to a local `systemd-journald` (journal namespaces supported), or to a remote `systemd-journal-remote`.
35
36
## Processing pipeline
37
@@ -44,19 +43,19 @@ The sequence of processing in Netdata's `log2journal` is designed to methodicall
43
2. **Extract Fields and Values**<br/>
44
Based on the input format (JSON, logfmt, or custom pattern), it extracts fields and their values from each log line. In the case of JSON and logfmt, it automatically extracts all fields. For custom patterns, it uses PCRE2 regular expressions, and fields are extracted based on sub-expressions defined in the pattern.
45
47
-3. **Transliteration**<br/>
46
+3. **Transliteration**<br/>
47
Extracted fields are transliterated to the limited character set accepted by systemd-journal: capitals A-Z, digits 0-9, underscores.
48
49
4. **Apply Optional Prefix**<br/>
50
If a prefix is specified, it is added to all keys. This happens before any other processing so that all subsequent matches and manipulations take the prefix into account.
51
53
-5. **Rename Fields**<br/>
52
+5. **Rename Fields**<br/>
53
Renames fields as specified in the configuration. This is used to change the names of the fields to match desired or required naming conventions.
54
55
6. **Inject New Fields**<br/>
56
New fields are injected into the log data. This can include constants or values derived from other fields, using variable substitution.
57
59
-7. **Rewrite Field Values**<br/>
58
+7. **Rewrite Field Values**<br/>
59
Applies rewriting rules to alter the values of the fields. This can involve complex transformations, including regular expressions and variable substitutions. The rewrite rules can also inject new fields into the data.
60
61
8. **Filter Fields**<br/>
@@ -81,7 +80,7 @@ We have an nginx server logging in this standard combined log format:
80
81
First, let's find the right pattern for `log2journal`. We ask ChatGPT:
82
84
-```
83
+```txt
84
My nginx log uses this log format:
85
86
log_format access '$remote_addr - $remote_user [$time_local] '
@@ -122,11 +121,11 @@ ChatGPT replies with this:
121
Let's see what the above says:
122
123
1. `(?x)`: enable PCRE2 extended mode. In this mode spaces and newlines in the pattern are ignored. To match a space you have to use `\s`. This mode allows us to split the pattern is multiple lines and add comments to it.
125
-1. `^`: match the beginning of the line
126
-2. `(?<remote_addr[^ ]+)`: match anything up to the first space (`[^ ]+`), and name it `remote_addr`.
127
-3. `\s`: match a space
128
-4. `-`: match a hyphen
129
-5. and so on...
124
+2. `^`: match the beginning of the line
125
+3. `(?<remote_addr[^ ]+)`: match anything up to the first space (`[^ ]+`), and name it `remote_addr`.
126
+4. `\s`: match a space
127
+5. `-`: match a hyphen
128
+6. and so on...
129
130
We edit `nginx.yaml` and add it, like this:
131
@@ -427,7 +426,6 @@ Rewrite rules are powerful. You can have named groups in them, like in the main
426
427
Now the message is ready to be sent to a systemd-journal. For this we use `systemd-cat-native`. This command can send such messages to a journal running on the localhost, a local journal namespace, or a `systemd-journal-remote` running on another server. By just appending `| systemd-cat-native` to the command, the message will be sent to the local journal.
428
430
-
429
```bash
430
# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 "-" "Go-http-client/1.1"' | log2journal -f nginx.yaml | systemd-cat-native
431
# no output
@@ -486,7 +484,7 @@ tail -F /var/log/nginx/access.log |\
484
485
Create the file `/etc/systemd/system/nginx-logs.service` (change `/path/to/nginx.yaml` to the right path):
486
489
-```
487
+```txt
488
[Unit]
489
Description=NGINX Log to Systemd Journal
490
After=network.target
@@ -524,7 +522,6 @@ Netdata will automatically pick the new namespace and present it at the list of
522
523
You can also instruct `systemd-cat-native` to log to a remote system, sending the logs to a `systemd-journal-remote` instance running on another server. Check [the manual of systemd-cat-native](/src/libnetdata/log/systemd-cat-native.md).
524
527
-
525
## Performance
526
527
`log2journal` and `systemd-cat-native` have been designed to process hundreds of thousands of log lines per second. They both utilize high performance indexing hashtables to speed up lookups, and queues that dynamically adapt to the number of log lines offered, offering a smooth and fast experience under all conditions.
@@ -537,15 +534,15 @@ The key characteristic that can influence the performance of a logs processing p
534
535
Especially the pattern `.*` seems to have the biggest impact on CPU consumption, especially when multiple `.*` are on the same pattern.
536
540
-Usually we use `.*` to indicate that we need to match everything up to a character, e.g. `.* ` to match up to a space. By replacing it with `[^ ]+` (meaning: match at least a character up to a space), the regular expression engine can be a lot more efficient, reducing the overall CPU utilization significantly.
537
+Usually we use `.*` to indicate that we need to match everything up to a character, e.g. `.*` to match up to a space. By replacing it with `[^ ]+` (meaning: match at least a character up to a space), the regular expression engine can be a lot more efficient, reducing the overall CPU utilization significantly.
538
539
### Performance of systemd journals
540
541
The ingestion pipeline of logs, from `tail` to `systemd-journald` or `systemd-journal-remote` is very efficient in all aspects. CPU utilization is better than any other system we tested and RAM usage is independent of the number of fields indexed, making systemd-journal one of the most efficient log management engines for ingesting high volumes of structured logs.
542
546
-High fields cardinality does not have a noticable impact on systemd-journal. The amount of fields indexed and the amount of unique values per field, have a linear and predictable result in the resource utilization of `systemd-journald` and `systemd-journal-remote`. This is unlike other logs management solutions, like Loki, that their RAM requirements grow exponentially as the cardinality increases, making it impractical for them to index the amount of information systemd journals can index.
543
+High fields cardinality does not have a noticeable impact on systemd-journal. The amount of fields indexed and the amount of unique values per field, have a linear and predictable result in the resource utilization of `systemd-journald` and `systemd-journal-remote`. This is unlike other logs management solutions, like Loki, that their RAM requirements grow exponentially as the cardinality increases, making it impractical for them to index the amount of information systemd journals can index.
544
548
-However, the number of fields added to journals influences the overall disk footprint. Less fields means more log entries per journal file, smaller overall disk footprint and faster queries.
545
+However, the number of fields added to journals influences the overall disk footprint. Less fields means more log entries per journal file, smaller overall disk footprint and faster queries.
546
547
systemd-journal files are primarily designed for security and reliability. This comes at the cost of disk footprint. The internal structure of journal files is such that in case of corruption, minimum data loss will incur. To achieve such a unique characteristic, certain data within the files need to be aligned at predefined boundaries, so that in case there is a corruption, non-corrupted parts of the journal file can be recovered.
548
@@ -578,7 +575,7 @@ If on other hand your organization prefers to maintain the full logs and control
575
576
## `log2journal` options
577
581
-```
578
+```txt
579
580
Netdata log2journal v1.43.0-341-gdac4df856
581
src/collectors/proc.plugin/README.md
+188
-191
@@ -6,35 +6,35 @@ This plugin is not an external plugin, but one of Netdata's threads.
6
7
In detail, it collects metrics from:
8
9
-- `/proc/net/dev` (all network interfaces for all their values)
10
-- `/proc/diskstats` (all disks for all their values)
11
-- `/proc/mdstat` (status of RAID arrays)
12
-- `/proc/net/snmp` (total IPv4, TCP and UDP usage)
13
-- `/proc/net/snmp6` (total IPv6 usage)
14
-- `/proc/net/netstat` (more IPv4 usage)
15
-- `/proc/net/wireless` (wireless extension)
16
-- `/proc/net/stat/nf_conntrack` (connection tracking performance)
17
-- `/proc/net/stat/synproxy` (synproxy performance)
18
-- `/proc/net/ip_vs/stats` (IPVS connection statistics)
19
-- `/proc/stat` (CPU utilization and attributes)
20
-- `/proc/meminfo` (memory information)
21
-- `/proc/vmstat` (system performance)
22
-- `/proc/net/rpc/nfsd` (NFS server statistics for both v3 and v4 NFS servers)
23
-- `/sys/fs/cgroup` (Control Groups - Linux Containers)
24
-- `/proc/self/mountinfo` (mount points)
25
-- `/proc/interrupts` (total and per core hardware interrupts)
26
-- `/proc/softirqs` (total and per core software interrupts)
27
-- `/proc/loadavg` (system load and total processes running)
28
-- `/proc/pressure/{cpu,memory,io}` (pressure stall information)
29
-- `/proc/sys/kernel/random/entropy_avail` (random numbers pool availability - used in cryptography)
30
-- `/proc/spl/kstat/zfs/arcstats` (status of ZFS adaptive replacement cache)
31
-- `/proc/spl/kstat/zfs/pool/state` (state of ZFS pools)
32
-- `/sys/class/power_supply` (power supply properties)
33
-- `/sys/class/infiniband` (infiniband interconnect)
34
-- `/sys/class/drm` (AMD GPUs)
35
-- `ipc` (IPC semaphores and message queues)
36
-- `ksm` Kernel Same-Page Merging performance (several files under `/sys/kernel/mm/ksm`).
37
-- `netdata` (internal Netdata resources utilization)
9
+- `/proc/net/dev` (all network interfaces for all their values)
10
+- `/proc/diskstats` (all disks for all their values)
11
+- `/proc/mdstat` (status of RAID arrays)
12
+- `/proc/net/snmp` (total IPv4, TCP and UDP usage)
13
+- `/proc/net/snmp6` (total IPv6 usage)
14
+- `/proc/net/netstat` (more IPv4 usage)
15
+- `/proc/net/wireless` (wireless extension)
16
+- `/proc/net/stat/nf_conntrack` (connection tracking performance)
17
+- `/proc/net/stat/synproxy` (synproxy performance)
18
+- `/proc/net/ip_vs/stats` (IPVS connection statistics)
19
+- `/proc/stat` (CPU utilization and attributes)
20
+- `/proc/meminfo` (memory information)
21
+- `/proc/vmstat` (system performance)
22
+- `/proc/net/rpc/nfsd` (NFS server statistics for both v3 and v4 NFS servers)
23
+- `/sys/fs/cgroup` (Control Groups - Linux Containers)
24
+- `/proc/self/mountinfo` (mount points)
25
+- `/proc/interrupts` (total and per core hardware interrupts)
26
+- `/proc/softirqs` (total and per core software interrupts)
27
+- `/proc/loadavg` (system load and total processes running)
28
+- `/proc/pressure/{cpu,memory,io}` (pressure stall information)
29
+- `/proc/sys/kernel/random/entropy_avail` (random numbers pool availability - used in cryptography)
30
+- `/proc/spl/kstat/zfs/arcstats` (status of ZFS adaptive replacement cache)
31
+- `/proc/spl/kstat/zfs/pool/state` (state of ZFS pools)
32
+- `/sys/class/power_supply` (power supply properties)
33
+- `/sys/class/infiniband` (infiniband interconnect)
34
+- `/sys/class/drm` (AMD GPUs)
35
+- `ipc` (IPC semaphores and message queues)
36
+- `ksm` Kernel Same-Page Merging performance (several files under `/sys/kernel/mm/ksm`).
37
+- `netdata` (internal Netdata resources utilization)
38
39
- - -
40
@@ -48,47 +48,47 @@ Hopefully, the Linux kernel provides many metrics that can provide deep insights
48
49
### Monitored disk metrics
50
51
-- **I/O bandwidth/s (kb/s)**
51
+- **I/O bandwidth/s (kb/s)**
52
The amount of data transferred from and to the disk.
53
-- **Amount of discarded data (kb/s)**
54
-- **I/O operations/s**
53
+- **Amount of discarded data (kb/s)**
54
+- **I/O operations/s**
55
The number of I/O operations completed.
56
-- **Extended I/O operations/s**
56
+- **Extended I/O operations/s**
57
The number of extended I/O operations completed.
58
-- **Queued I/O operations**
58
+- **Queued I/O operations**
59
The number of currently queued I/O operations. For traditional disks that execute commands one after another, one of them is being run by the disk and the rest are just waiting in a queue.
60
-- **Backlog size (time in ms)**
60
+- **Backlog size (time in ms)**
61
The expected duration of the currently queued I/O operations.
62
-- **Utilization (time percentage)**
62
+- **Utilization (time percentage)**
63
The percentage of time the disk was busy with something. This is a very interesting metric, since for most disks, that execute commands sequentially, **this is the key indication of congestion**. A sequential disk that is 100% of the available time busy, has no time to do anything more, so even if the bandwidth or the number of operations executed by the disk is low, its capacity has been reached.
64
Of course, for newer disk technologies (like fusion cards) that are capable to execute multiple commands in parallel, this metric is just meaningless.
65
-- **Average I/O operation time (ms)**
65
+- **Average I/O operation time (ms)**
66
The average time for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
67
-- **Average I/O operation time for extended operations (ms)**
67
+- **Average I/O operation time for extended operations (ms)**
68
The average time for extended I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
69
-- **Average I/O operation size (kb)**
69
+- **Average I/O operation size (kb)**
70
The average amount of data of the completed I/O operations.
71
-- **Average amount of discarded data (kb)**
71
+- **Average amount of discarded data (kb)**
72
The average amount of data of the completed discard operations.
73
-- **Average Service Time (ms)**
73
+- **Average Service Time (ms)**
74
The average service time for completed I/O operations. This metric is calculated using the total busy time of the disk and the number of completed operations. If the disk is able to execute multiple parallel operations the reporting average service time will be misleading.
75
-- **Average Service Time for extended I/O operations (ms)**
75
+- **Average Service Time for extended I/O operations (ms)**
76
The average service time for completed extended I/O operations.
77
-- **Merged I/O operations/s**
77
+- **Merged I/O operations/s**
78
The Linux kernel is capable of merging I/O operations. So, if two requests to read data from the disk are adjacent, the Linux kernel may merge them to one before giving them to disk. This metric measures the number of operations that have been merged by the Linux kernel.
79
-- **Merged discard operations/s**
80
-- **Total I/O time**
79
+- **Merged discard operations/s**
80
+- **Total I/O time**
81
The sum of the duration of all completed I/O operations. This number can exceed the interval if the disk is able to execute multiple I/O operations in parallel.
82
-- **Space usage**
82
+- **Space usage**
83
For mounted disks, Netdata will provide a chart for their space, with 3 dimensions:
84
- 1. free
85
- 2. used
86
- 3. reserved for root
87
-- **inode usage**
84
+ 1. free
85
+ 2. used
86
+ 3. reserved for root
87
+- **inode usage**
88
For mounted disks, Netdata will provide a chart for their inodes (number of file and directories), with 3 dimensions:
89
- 1. free
90
- 2. used
91
- 3. reserved for root
89
+ 1. free
90
+ 2. used
91
+ 3. reserved for root
92
93
### disk names
94
@@ -100,9 +100,9 @@ By default, Netdata will enable monitoring metrics only when they are not zero.
100
101
Netdata categorizes all block devices in 3 categories:
102
103
-1. physical disks (i.e. block devices that do not have child devices and are not partitions)
104
-2. virtual disks (i.e. block devices that have child devices - like RAID devices)
105
-3. disk partitions (i.e. block devices that are part of a physical disk)
103
+1. physical disks (i.e. block devices that do not have child devices and are not partitions)
104
+2. virtual disks (i.e. block devices that have child devices - like RAID devices)
105
+3. disk partitions (i.e. block devices that are part of a physical disk)
106
107
Performance metrics are enabled by default for all disk devices, except partitions and not-mounted virtual disks. Of course, you can enable/disable monitoring any block device by editing the Netdata configuration file.
108
@@ -118,7 +118,7 @@ mv netdata.conf.new netdata.conf
118
119
Then edit `netdata.conf` and find the following section. This is the basic plugin configuration.
120
121
-```
121
+```txt
122
[plugin:proc:/proc/diskstats]
123
# enable new disks detected at runtime = yes
124
# performance metrics for physical disks = auto
@@ -152,25 +152,25 @@ Then edit `netdata.conf` and find the following section. This is the basic plugi
152
153
For each virtual disk, physical disk and partition you will have a section like this:
154
155
-```
155
+```txt
156
[plugin:proc:/proc/diskstats:sda]
157
- # enable = yes
158
- # enable performance metrics = auto
159
- # bandwidth = auto
160
- # operations = auto
161
- # merged operations = auto
162
- # i/o time = auto
163
- # queued operations = auto
164
- # utilization percentage = auto
157
+ # enable = yes
158
+ # enable performance metrics = auto
159
+ # bandwidth = auto
160
+ # operations = auto
161
+ # merged operations = auto
162
+ # i/o time = auto
163
+ # queued operations = auto
164
+ # utilization percentage = auto
165
# extended operations = auto
166
- # backlog = auto
166
+ # backlog = auto
167
```
168
169
For all configuration options:
170
171
-- `auto` = enable monitoring if the collected values are not zero
172
-- `yes` = enable monitoring
173
-- `no` = disable monitoring
171
+- `auto` = enable monitoring if the collected values are not zero
172
+- `yes` = enable monitoring
173
+- `no` = disable monitoring
174
175
Of course, to set options, you will have to uncomment them. The comments show the internal defaults.
176
@@ -180,14 +180,14 @@ After saving `/etc/netdata/netdata.conf`, restart your Netdata to apply them.
180
181
You can pretty easy disable performance metrics for individual device, for ex.:
182
183
-```
183
+```txt
184
[plugin:proc:/proc/diskstats:sda]
185
- enable performance metrics = no
185
+ enable performance metrics = no
186
```
187
188
But sometimes you need disable performance metrics for all devices with the same type, to do it you need to figure out device type from `/proc/diskstats` for ex.:
189
190
-```
190
+```txt
191
7 0 loop0 1651 0 3452 168 0 0 0 0 0 8 168
192
7 1 loop1 4955 0 11924 880 0 0 0 0 0 64 880
193
7 2 loop2 36 0 216 4 0 0 0 0 0 4 4
@@ -200,7 +200,7 @@ But sometimes you need disable performance metrics for all devices with the same
200
All zram devices starts with `251` number and all loop devices starts with `7`.
201
So, to disable performance metrics for all loop devices you could add `performance metrics for disks with major 7 = no` to `[plugin:proc:/proc/diskstats]` section.
202
203
-```
203
+```txt
204
[plugin:proc:/proc/diskstats]
205
performance metrics for disks with major 7 = no
206
```
@@ -209,34 +209,34 @@ So, to disable performance metrics for all loop devices you could add `performan
209
210
### Monitored RAID array metrics
211
212
-1. **Health** Number of failed disks in every array (aggregate chart).
212
+1. **Health** Number of failed disks in every array (aggregate chart).
213
214
-2. **Disks stats**
214
+2. **Disks stats**
215
216
-- total (number of devices array ideally would have)
217
-- inuse (number of devices currently are in use)
216
+ - total (number of devices array ideally would have)
217
+ - inuse (number of devices currently are in use)
218
219
-3. **Mismatch count**
219
+3. **Mismatch count**
220
221
-- unsynchronized blocks
221
+ - unsynchronized blocks
222
223
-4. **Current status**
223
+4. **Current status**
224
225
-- resync in percent
226
-- recovery in percent
227
-- reshape in percent
228
-- check in percent
225
+ - resync in percent
226
+ - recovery in percent
227
+ - reshape in percent
228
+ - check in percent
229
230
-5. **Operation status** (if resync/recovery/reshape/check is active)
230
+5. **Operation status** (if resync/recovery/reshape/check is active)
231
232
-- finish in minutes
233
-- speed in megabytes/s
232
+ - finish in minutes
233
+ - speed in megabytes/s
234
235
-6. **Nonredundant array availability**
235
+6. **Non-redundant array availability**
236
237
#### configuration
238
239
-```
239
+```txt
240
[plugin:proc:/proc/mdstat]
241
# faulty devices = yes
242
# nonredundant arrays availability = yes
@@ -311,50 +311,50 @@ each state.
311
312
### Monitored memory metrics
313
314
-- Amount of memory swapped in/out
315
-- Amount of memory paged from/to disk
316
-- Number of memory page faults
317
-- Number of out of memory kills
318
-- Number of NUMA events
314
+- Amount of memory swapped in/out
315
+- Amount of memory paged from/to disk
316
+- Number of memory page faults
317
+- Number of out of memory kills
318
+- Number of NUMA events
319
320
### Configuration
321
322
```conf
323
[plugin:proc:/proc/vmstat]
324
- filename to monitor = /proc/vmstat
325
- swap i/o = auto
326
- disk i/o = yes
327
- memory page faults = yes
328
- out of memory kills = yes
329
- system-wide numa metric summary = auto
324
+ filename to monitor = /proc/vmstat
325
+ swap i/o = auto
326
+ disk i/o = yes
327
+ memory page faults = yes
328
+ out of memory kills = yes
329
+ system-wide numa metric summary = auto
330
```
331
332
## Monitoring Network Interfaces
333
334
### Monitored network interface metrics
335
336
-- **Physical Network Interfaces Aggregated Bandwidth (kilobits/s)**
336
+- **Physical Network Interfaces Aggregated Bandwidth (kilobits/s)**
337
The amount of data received and sent through all physical interfaces in the system. This is the source of data for the Net Inbound and Net Outbound dials in the System Overview section.
338
339
-- **Bandwidth (kilobits/s)**
339
+- **Bandwidth (kilobits/s)**
340
The amount of data received and sent through the interface.
341
342
-- **Packets (packets/s)**
342
+- **Packets (packets/s)**
343
The number of packets received, packets sent, and multicast packets transmitted through the interface.
344
345
-- **Interface Errors (errors/s)**
345
+- **Interface Errors (errors/s)**
346
The number of errors for the inbound and outbound traffic on the interface.
347
348
-- **Interface Drops (drops/s)**
348
+- **Interface Drops (drops/s)**
349
The number of packets dropped for the inbound and outbound traffic on the interface.
350
351
-- **Interface FIFO Buffer Errors (errors/s)**
351
+- **Interface FIFO Buffer Errors (errors/s)**
352
The number of FIFO buffer errors encountered while receiving and transmitting data through the interface.
353
354
-- **Compressed Packets (packets/s)**
354
+- **Compressed Packets (packets/s)**
355
The number of compressed packets transmitted or received by the device driver.
356
357
-- **Network Interface Events (events/s)**
357
+- **Network Interface Events (events/s)**
358
The number of packet framing errors, collisions detected on the interface, and carrier losses detected by the device driver.
359
360
By default Netdata will enable monitoring metrics only when they are not zero. If they are constantly zero they are ignored. Metrics that will start having values, after Netdata is started, will be detected and charts will be automatically added to the dashboard (a refresh of the dashboard is needed for them to appear though).
@@ -372,43 +372,43 @@ The settings for monitoring wireless is in the `[plugin:proc:/proc/net/wireless]
372
373
You can set the following values for each configuration option:
374
375
-- `auto` = enable monitoring if the collected values are not zero
376
-- `yes` = enable monitoring
377
-- `no` = disable monitoring
375
+- `auto` = enable monitoring if the collected values are not zero
376
+- `yes` = enable monitoring
377
+- `no` = disable monitoring
378
379
#### Monitored wireless interface metrics
380
381
-- **Status**
381
+- **Status**
382
The current state of the interface. This is a device-dependent option.
383
384
-- **Link**
385
- Overall quality of the link.
384
+- **Link**
385
+ Overall quality of the link.
386
387
-- **Level**
387
+- **Level**
388
Received signal strength (RSSI), which indicates how strong the received signal is.
389
-
390
-- **Noise**
391
- Background noise level.
392
-
393
-- **Discarded packets**
394
- Discarded packets for: Number of packets received with a different NWID or ESSID (`nwid`), unable to decrypt (`crypt`), hardware was not able to properly re-assemble the link layer fragments (`frag`), packets failed to deliver (`retry`), and packets lost in relation with specific wireless operations (`misc`).
395
-
396
-- **Missed beacon**
389
+
390
+- **Noise**
391
+ Background noise level.
392
+
393
+- **Discarded packets**
394
+ Discarded packets for: Number of packets received with a different NWID or ESSID (`nwid`), unable to decrypt (`crypt`), hardware was not able to properly re-assemble the link layer fragments (`frag`), packets failed to deliver (`retry`), and packets lost in relation with specific wireless operations (`misc`).
395
+
396
+- **Missed beacon**
397
Number of periodic beacons from the cell or the access point the interface has missed.
398
-
399
-#### Wireless configuration
398
+
399
+#### Wireless configuration
400
401
#### alerts
402
403
There are several alerts defined in `health.d/net.conf`.
404
405
-The tricky ones are `inbound packets dropped` and `inbound packets dropped ratio`. They have quite a strict policy so that they warn users about possible issues. These alerts can be annoying for some network configurations. It is especially true for some bonding configurations if an interface is a child or a bonding interface itself. If it is expected to have a certain number of drops on an interface for a certain network configuration, a separate alert with different triggering thresholds can be created or the existing one can be disabled for this specific interface. It can be done with the help of the [families](/src/health/REFERENCE.md#alert-line-families) line in the alert configuration. For example, if you want to disable the `inbound packets dropped` alert for `eth0`, set `families: !eth0 *` in the alert definition for `template: inbound_packets_dropped`.
405
+The tricky ones are `inbound packets dropped` and `inbound packets dropped ratio`. They have quite a strict policy so that they warn users about possible issues. These alerts can be annoying for some network configurations. It is especially true for some bonding configurations if an interface is a child or a bonding interface itself. If it is expected to have a certain number of drops on an interface for a certain network configuration, a separate alert with different triggering thresholds can be created or the existing one can be disabled for this specific interface. It can be done with the help of the families line in the alert configuration. For example, if you want to disable the `inbound packets dropped` alert for `eth0`, set `families: !eth0 *` in the alert definition for `template: inbound_packets_dropped`.
406
407
#### configuration
408
409
Module configuration:
410
411
-```
411
+```txt
412
[plugin:proc:/proc/net/dev]
413
# filename to monitor = /proc/net/dev
414
# path to get virtual interfaces = /sys/devices/virtual/net/%s
@@ -427,7 +427,7 @@ Module configuration:
427
428
Per interface configuration:
429
430
-```
430
+```txt
431
[plugin:proc:/proc/net/dev:enp0s3]
432
# enabled = yes
433
# virtual = no
@@ -444,8 +444,6 @@ Per interface configuration:
444
445
446
447
----
448
-
447
SYNPROXY is a TCP SYN packets proxy. It can be used to protect any TCP server (like a web server) from SYN floods and similar DDos attacks.
448
449
SYNPROXY is a netfilter module, in the Linux kernel (since version 3.12). It is optimized to handle millions of packets per second utilizing all CPUs available without any concurrency locking between the connections.
@@ -454,8 +452,8 @@ The net effect of this, is that the real servers will not notice any change duri
452
453
Netdata does not enable SYNPROXY. It just uses the SYNPROXY metrics exposed by your kernel, so you will first need to configure it. The hard way is to run iptables SYNPROXY commands directly on the console. An easier way is to use [FireHOL](https://firehol.org/), which, is a firewall manager for iptables. FireHOL can configure SYNPROXY using the following setup guides:
454
457
-- **[Working with SYNPROXY](https://github.com/firehol/firehol/wiki/Working-with-SYNPROXY)**
458
-- **[Working with SYNPROXY and traps](https://github.com/firehol/firehol/wiki/Working-with-SYNPROXY-and-traps)**
455
+- **[Working with SYNPROXY](https://github.com/firehol/firehol/wiki/Working-with-SYNPROXY)**
456
+- **[Working with SYNPROXY and traps](https://github.com/firehol/firehol/wiki/Working-with-SYNPROXY-and-traps)**
457
458
### Real-time monitoring of Linux Anti-DDoS
459
@@ -463,10 +461,10 @@ Netdata is able to monitor in real-time (per second updates) the operation of th
461
462
It visualizes 4 charts:
463
466
-1. TCP SYN Packets received on ports operated by SYNPROXY
467
-2. TCP Cookies (valid, invalid, retransmits)
468
-3. Connections Reopened
469
-4. Entries used
464
+1. TCP SYN Packets received on ports operated by SYNPROXY
465
+2. TCP Cookies (valid, invalid, retransmits)
466
+3. Connections Reopened
467
+4. Entries used
468
469
Example image:
470
@@ -483,37 +481,37 @@ battery capacity.
481
Depending on the underlying driver, it may provide the following charts
482
and metrics:
483
486
-1. Capacity: The power supply capacity expressed as a percentage.
484
+1. Capacity: The power supply capacity expressed as a percentage.
485
488
- - capacity_now
486
+ - capacity_now
487
490
-2. Charge: The charge for the power supply, expressed as amphours.
488
+2. Charge: The charge for the power supply, expressed as amp-hours.
489
492
- - charge_full_design
493
- - charge_full
494
- - charge_now
495
- - charge_empty
496
- - charge_empty_design
490
+ - charge_full_design
491
+ - charge_full
492
+ - charge_now
493
+ - charge_empty
494
+ - charge_empty_design
495
498
-3. Energy: The energy for the power supply, expressed as watthours.
496
+3. Energy: The energy for the power supply, expressed as watthours.
497
500
- - energy_full_design
501
- - energy_full
502
- - energy_now
503
- - energy_empty
504
- - energy_empty_design
498
+ - energy_full_design
499
+ - energy_full
500
+ - energy_now
501
+ - energy_empty
502
+ - energy_empty_design
503
506
-4. Voltage: The voltage for the power supply, expressed as volts.
504
+4. Voltage: The voltage for the power supply, expressed as volts.
505
508
- - voltage_max_design
509
- - voltage_max
510
- - voltage_now
511
- - voltage_min
512
- - voltage_min_design
506
+ - voltage_max_design
507
+ - voltage_max
508
+ - voltage_now
509
+ - voltage_min
510
+ - voltage_min_design
511
514
-#### configuration
512
+### configuration
513
516
-```
514
+```txt
515
[plugin:proc:/sys/class/power_supply]
516
# battery capacity = yes
517
# battery charge = no
@@ -524,18 +522,18 @@ and metrics:
522
# directory to monitor = /sys/class/power_supply
523
```
524
527
-#### notes
525
+### notes
526
529
-- Most drivers provide at least the first chart. Battery powered ACPI
527
+- Most drivers provide at least the first chart. Battery powered ACPI
528
compliant systems (like most laptops) provide all but the third, but do
529
not provide all of the metrics for each chart.
530
533
-- Current, energy, and voltages are reported with a *very* high precision
531
+- Current, energy, and voltages are reported with a *very* high precision
532
by the power_supply framework. Usually, this is far higher than the
533
actual hardware supports reporting, so expect to see changes in these
534
charts jump instead of scaling smoothly.
535
538
-- If `max` or `full` attribute is defined by the driver, but not a
536
+- If `max` or `full` attribute is defined by the driver, but not a
537
corresponding `min` or `empty` attribute, then Netdata will still provide
538
the corresponding `min` or `empty`, which will then always read as zero.
539
This way, alerts which match on these will still work.
@@ -548,17 +546,17 @@ This module monitors every active Infiniband port. It provides generic counters
546
547
Each port will have its counters metrics monitored, grouped in the following charts:
548
551
-- **Bandwidth usage**
549
+- **Bandwidth usage**
550
Sent/Received data, in KB/s
551
554
-- **Packets Statistics**
552
+- **Packets Statistics**
553
Sent/Received packets, in 3 categories: total, unicast and multicast.
554
557
-- **Errors Statistics**
555
+- **Errors Statistics**
556
Many errors counters are provided, presenting statistics for:
559
- - Packets: malformed, sent/received discarded by card/switch, missing resource
560
- - Link: downed, recovered, integrity error, minor error
561
- - Other events: Tick Wait to send, buffer overrun
557
+ - Packets: malformed, sent/received discarded by card/switch, missing resource
558
+ - Link: downed, recovered, integrity error, minor error
559
+ - Other events: Tick Wait to send, buffer overrun
560
561
If your vendor is supported, you'll also get HW-Counters statistics. These being vendor specific, please refer to their documentation.
562
@@ -568,7 +566,7 @@ If your vendor is supported, you'll also get HW-Counters statistics. These being
566
567
Default configuration will monitor only enabled infiniband ports, and refresh newly activated or created ports every 30 seconds
568
571
-```
569
+```txt
570
[plugin:proc:/sys/class/infiniband]
571
# dirname to monitor = /sys/class/infiniband
572
# bandwidth counters = yes
@@ -589,45 +587,46 @@ This module monitors every AMD GPU card discovered at agent startup.
587
588
The following charts will be provided:
589
592
-- **GPU utilization**
593
-- **GPU memory utilization**
594
-- **GPU clock frequency**
595
-- **GPU memory clock frequency**
596
-- **VRAM memory usage percentage**
597
-- **VRAM memory usage**
598
-- **visible VRAM memory usage percentage**
599
-- **visible VRAM memory usage**
600
-- **GTT memory usage percentage**
601
-- **GTT memory usage**
590
+- **GPU utilization**
591
+- **GPU memory utilization**
592
+- **GPU clock frequency**
593
+- **GPU memory clock frequency**
594
+- **VRAM memory usage percentage**
595
+- **VRAM memory usage**
596
+- **visible VRAM memory usage percentage**
597
+- **visible VRAM memory usage**
598
+- **GTT memory usage percentage**
599
+- **GTT memory usage**
600
601
### configuration
602
603
The `drm` path can be configured if it differs from the default:
604
607
-```
605
+```txt
606
[plugin:proc:/sys/class/drm]
607
# directory to monitor = /sys/class/drm
608
```
609
612
-> [!NOTE]
610
+> **Note**
611
+>
612
> Temperature, fan speed, voltage and power metrics for AMD GPUs can be monitored using the [Sensors](/src/go/plugin/go.d/modules/sensors/README.md) plugin.
613
614
## IPC
615
616
### Monitored IPC metrics
617
619
-- **number of messages in message queues**
620
-- **amount of memory used by message queues**
621
-- **number of semaphores**
622
-- **number of semaphore arrays**
623
-- **number of shared memory segments**
624
-- **amount of memory used by shared memory segments**
618
+- **number of messages in message queues**
619
+- **amount of memory used by message queues**
620
+- **number of semaphores**
621
+- **number of semaphore arrays**
622
+- **number of shared memory segments**
623
+- **amount of memory used by shared memory segments**
624
625
As far as the message queue charts are dynamic, sane limits are applied for the number of dimensions per chart (the limit is configurable).
626
627
### configuration
628
630
-```
629
+```txt
630
[plugin:proc:ipc]
631
# message queues = yes
632
# semaphore totals = yes
@@ -636,5 +635,3 @@ As far as the message queue charts are dynamic, sane limits are applied for the
635
# shm filename to monitor = /proc/sysvipc/shm
636
# max dimensions in memory allowed = 50
637
```
639
-
640
-
src/collectors/profile.plugin/README.md
+6
-6
@@ -4,11 +4,11 @@ This plugin allows someone to backfill an agent with random data.
4
5
A user can specify:
6
7
- - The number charts they want,
8
- - the number of dimensions per chart,
9
- - the desire update every collection frequency,
10
- - the number of seconds to backfill.
11
- - the number of collection threads.
7
+- The number charts they want,
8
+- the number of dimensions per chart,
9
+- the desire update every collection frequency,
10
+- the number of seconds to backfill.
11
+- the number of collection threads.
12
13
## Configuration
14
@@ -16,7 +16,7 @@ Edit the `netdata.conf` configuration file using [`edit-config`](/docs/netdata-a
16
17
Scroll down to the `[plugin:profile]` section to find the available options:
18
19
-```
19
+```txt
20
[plugin:profile]
21
update every = 5
22
number of charts = 200
src/collectors/python.d.plugin/README.md
+11
-20
@@ -1,22 +1,13 @@
1
-<!--
2
-title: "python.d.plugin"
3
-custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/README.md"
4
-sidebar_label: "python.d.plugin"
5
-learn_status: "Published"
6
-learn_topic_type: "Tasks"
7
-learn_rel_path: "Developers/External plugins/python.d.plugin"
8
--->
9
-
1
# python.d.plugin
2
3
`python.d.plugin` is a Netdata external plugin. It is an **orchestrator** for data collection modules written in `python`.
4
14
-1. It runs as an independent process `ps fax` shows it
15
-2. It is started and stopped automatically by Netdata
16
-3. It communicates with Netdata via a unidirectional pipe (sending data to the `netdata` daemon)
17
-4. Supports any number of data collection **modules**
18
-5. Allows each **module** to have one or more data collection **jobs**
19
-6. Each **job** is collecting one or more metrics from a single data source
5
+1. It runs as an independent process `ps fax` shows it
6
+2. It is started and stopped automatically by Netdata
7
+3. It communicates with Netdata via a unidirectional pipe (sending data to the `netdata` daemon)
8
+4. Supports any number of data collection **modules**
9
+5. Allows each **module** to have one or more data collection **jobs**
10
+6. Each **job** is collecting one or more metrics from a single data source
11
12
## Disclaimer
13
@@ -25,7 +16,7 @@ Module configurations are written in YAML and **pyYAML is required**.
16
17
Every configuration file must have one of two formats:
18
28
-- Configuration for only one job:
19
+- Configuration for only one job:
20
21
```yaml
22
update_every : 2 # update frequency
@@ -35,7 +26,7 @@ other_var1 : bla # variables passed to module
26
other_var2 : alb
27
```
28
38
-- Configuration for many jobs (ex. mysql):
29
+- Configuration for many jobs (ex. mysql):
30
31
```yaml
32
# module defaults:
@@ -55,19 +46,19 @@ other_job:
46
47
## How to debug a python module
48
58
-```
49
+```bash
50
# become user netdata
51
sudo su -s /bin/bash netdata
52
```
53
54
Depending on where Netdata was installed, execute one of the following commands to trace the execution of a python module:
55
65
-```
56
+```bash
57
# execute the plugin in debug mode, for a specific module
58
/opt/netdata/usr/libexec/netdata/plugins.d/python.d.plugin <module> debug trace
59
/usr/libexec/netdata/plugins.d/python.d.plugin <module> debug trace
60
```
61
71
-Where `[module]` is the directory name under <https://github.com/netdata/netdata/tree/master/src/collectors/python.d.plugin>
62
+Where `[module]` is the directory name under <https://github.com/netdata/netdata/tree/master/src/collectors/python.d.plugin>
63
64
**Note**: If you would like execute a collector in debug mode while it is still running by Netdata, you can pass the `nolock` CLI option to the above commands.
src/collectors/systemd-journal.plugin/README.md
+3
-4
@@ -1,4 +1,3 @@
1
-
1
# `systemd` journal plugin
2
3
[KEY FEATURES](#key-features) | [JOURNAL SOURCES](#journal-sources) | [JOURNAL FIELDS](#journal-fields) |
@@ -40,8 +39,8 @@ For more information check [this discussion](https://github.com/netdata/netdata/
39
40
The following are limitations related to the availability of the plugin:
41
43
-- Netdata versions prior to 1.44 shipped in a docker container do not include this plugin.
44
- The problem is that `libsystemd` is not available in Alpine Linux (there is a `libsystemd`, but it is a dummy that
42
+- Netdata versions prior to 1.44 shipped in a docker container do not include this plugin.
43
+ The problem is that `libsystemd` is not available in Alpine Linux (there is a `libsystemd`, but it is a dummy that
44
returns failure on all calls). Starting with Netdata version 1.44, Netdata containers use a Debian base image
45
making this plugin available when Netdata is running in a container.
46
- For the same reason (lack of `systemd` support for Alpine Linux), the plugin is not available on `static` builds of
@@ -321,7 +320,7 @@ algorithm to allow it respond promptly. It works like this:
320
6. In systemd versions 254 or later, the plugin fetches the unique sequence number of each log entry and calculates the
321
the percentage of the file matched by the query, versus the total number of the log entries in the journal file.
322
7. In systemd versions prior to 254, the plugin estimates the number of entries the journal file contributes to the
324
- query, using the amount of log entries matched it vs. the total duration the log file has entries for.
323
+ query, using the amount of log entries matched it vs. the total duration the log file has entries for.
324
325
The above allow the plugin to respond promptly even when the number of log entries in the journal files is several
326
dozens millions, while providing accurate estimations of the log entries over time at the histogram and enough counters
src/collectors/systemd-journal.plugin/active_journal_centralization_guide_no_encryption.md
+1
-1
@@ -47,7 +47,7 @@ sudo systemctl enable --now systemd-journal-gatewayd.socket
47
48
To use it, open your web browser and navigate to:
49
50
-```
50
+```txt
51
http://server.ip:19531/browse
52
```
53
src/collectors/systemd-journal.plugin/forward_secure_sealing.md
+6
-1
@@ -5,12 +5,14 @@ Given that attackers often try to hide their actions by modifying or deleting lo
5
FSS provides administrators with a mechanism to identify any such unauthorized alterations.
6
7
## Importance
8
+
9
Logs are a crucial component of system monitoring and auditing. Ensuring their integrity means administrators can trust
10
the data, detect potential breaches, and trace actions back to their origins. Traditional methods to maintain this
11
integrity involve writing logs to external systems or printing them out. While these methods are effective, they are
12
not foolproof. FSS offers a more streamlined approach, allowing for log verification directly on the local system.
13
14
## How FSS Works
15
+
16
FSS operates by "sealing" binary logs at regular intervals. This seal is a cryptographic operation, ensuring that any
17
tampering with the logs prior to the sealing can be detected. If an attacker modifies logs before they are sealed,
18
these changes become a permanent part of the sealed record, highlighting any malicious activity.
@@ -29,6 +31,7 @@ administrators to verify older seals. If logs are tampered with, verification wi
31
breach.
32
33
## Enabling FSS
34
+
35
To enable FSS, use the following command:
36
37
```bash
@@ -43,6 +46,7 @@ journalctl --setup-keys --interval=10s
46
```
47
48
## Verifying Journals
49
+
50
After enabling FSS, you can verify the integrity of your logs using the verification key:
51
52
```bash
@@ -52,6 +56,7 @@ journalctl --verify
56
If any discrepancies are found, you'll be alerted, indicating potential tampering.
57
58
## Disabling FSS
59
+
60
Should you wish to disable FSS:
61
62
**Delete the Sealing Key**: This stops new log entries from being sealed.
@@ -66,7 +71,6 @@ journalctl --rotate
71
journalctl --vacuum-time=1s
72
```
73
69
-
74
**Adjust Systemd Configuration (Optional)**: If you've made changes to facilitate FSS in `/etc/systemd/journald.conf`,
75
consider reverting or adjusting those. Restart the systemd-journald service afterward:
76
@@ -75,6 +79,7 @@ systemctl restart systemd-journald
79
```
80
81
## Conclusion
82
+
83
FSS is a significant advancement in maintaining log integrity. While not a replacement for all traditional integrity
84
methods, it offers a valuable tool in the battle against unauthorized log tampering. By integrating FSS into your log
85
management strategy, you ensure a more transparent, reliable, and tamper-evident logging system.
src/collectors/systemd-journal.plugin/passive_journal_centralization_guide_self_signed_certs.md
+3
-4
@@ -46,9 +46,9 @@ sudo ./systemd-journal-self-signed-certs.sh "server1" "DNS:hostname1" "IP:10.0.0
46
47
Where:
48
49
- - `server1` is the canonical name of the server. On newer systemd version, this name will be used by `systemd-journal-remote` and Netdata when you view the logs on the dashboard.
50
- - `DNS:hostname1` is a DNS name that the server is reachable at. Add `"DNS:xyz"` multiple times to define multiple DNS names for the server.
51
- - `IP:10.0.0.1` is an IP that the server is reachable at. Add `"IP:xyz"` multiple times to define multiple IPs for the server.
49
+- `server1` is the canonical name of the server. On newer systemd version, this name will be used by `systemd-journal-remote` and Netdata when you view the logs on the dashboard.
50
+- `DNS:hostname1` is a DNS name that the server is reachable at. Add `"DNS:xyz"` multiple times to define multiple DNS names for the server.
51
+- `IP:10.0.0.1` is an IP that the server is reachable at. Add `"IP:xyz"` multiple times to define multiple IPs for the server.
52
53
Repeat this process to create the certificates for all your servers. You can add servers as required, at any time in the future.
54
@@ -198,7 +198,6 @@ Here it is in action, in Netdata:
198
199
200
201
-
201
## Verify it works
202
203
To verify the central server is receiving logs, run this on the central server:
src/daemon/README.md
+29
-33
@@ -1,8 +1,8 @@
1
# Netdata daemon
2
3
-The Netdata daemon is practically a synonym for the Netdata Agent, as it controls its
4
-entire operation. We support various methods to
5
-[start, stop, or restart the daemon](/packaging/installer/README.md#maintaining-a-netdata-agent-installation).
3
+The Netdata daemon is practically a synonym for the Netdata Agent, as it controls its
4
+entire operation. We support various methods to
5
+[start, stop, or restart the daemon](/docs/netdata-agent/start-stop-restart.md).
6
7
This document provides some basic information on the command line options, log files, and how to debug and troubleshoot
8
@@ -116,10 +116,10 @@ You can send commands during runtime via [netdatacli](/src/cli/README.md).
116
117
Netdata uses 4 log files:
118
119
-1. `error.log`
120
-2. `collector.log`
121
-3. `access.log`
122
-4. `debug.log`
119
+1. `error.log`
120
+2. `collector.log`
121
+3. `access.log`
122
+4. `debug.log`
123
124
Any of them can be disabled by setting it to `/dev/null` or `none` in `netdata.conf`. By default `error.log`,
125
`collector.log`, and `access.log` are enabled. `debug.log` is only enabled if debugging/tracing is also enabled
@@ -133,8 +133,8 @@ The `error.log` is the `stderr` of the `netdata` daemon .
133
134
For most Netdata programs (including standard external plugins shipped by netdata), the following lines may appear:
135
136
-| tag | description |
137
-|:-:|:----------|
136
+| tag | description |
137
+|:-------:|:--------------------------------------------------------------------------------------------------------------------------|
138
| `INFO` | Something important the user should know. |
139
| `ERROR` | Something that might disable a part of netdata.<br/>The log line includes `errno` (if it is not zero). |
140
| `FATAL` | Something prevented a program from running.<br/>The log line includes `errno` (if it is not zero) and the program exited. |
@@ -166,15 +166,15 @@ DATE: ID: (sent/all = SENT_BYTES/ALL_BYTES bytes PERCENT_COMPRESSION%, prep/sent
166
167
where:
168
169
-- `ID` is the client ID. Client IDs are auto-incremented every time a client connects to netdata.
170
-- `SENT_BYTES` is the number of bytes sent to the client, without the HTTP response header.
171
-- `ALL_BYTES` is the number of bytes of the response, before compression.
172
-- `PERCENT_COMPRESSION` is the percentage of traffic saved due to compression.
173
-- `PREP_TIME` is the time in milliseconds needed to prepared the response.
174
-- `SENT_TIME` is the time in milliseconds needed to sent the response to the client.
175
-- `TOTAL_TIME` is the total time the request was inside Netdata (from the first byte of the request to the last byte
169
+- `ID` is the client ID. Client IDs are auto-incremented every time a client connects to netdata.
170
+- `SENT_BYTES` is the number of bytes sent to the client, without the HTTP response header.
171
+- `ALL_BYTES` is the number of bytes of the response, before compression.
172
+- `PERCENT_COMPRESSION` is the percentage of traffic saved due to compression.
173
+- `PREP_TIME` is the time in milliseconds needed to prepared the response.
174
+- `SENT_TIME` is the time in milliseconds needed to sent the response to the client.
175
+- `TOTAL_TIME` is the total time the request was inside Netdata (from the first byte of the request to the last byte
176
of the response).
177
-- `ACTION` can be `filecopy`, `options` (used in CORS), `data` (API call).
177
+- `ACTION` can be `filecopy`, `options` (used in CORS), `data` (API call).
178
179
### debug.log
180
@@ -198,13 +198,13 @@ You can set Netdata scheduling policy in `netdata.conf`, like this:
198
199
You can use the following:
200
201
-| policy | description |
202
-| :-----------------------: | :---------- |
203
-| `idle` | use CPU only when there is spare - this is lower than nice 19 - it is the default for Netdata and it is so low that Netdata will run in "slow motion" under extreme system load, resulting in short (1-2 seconds) gaps at the charts. |
201
+| policy | description |
202
+|:-------------------------:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
203
+| `idle` | use CPU only when there is spare - this is lower than nice 19 - it is the default for Netdata and it is so low that Netdata will run in "slow motion" under extreme system load, resulting in short (1-2 seconds) gaps at the charts. |
204
| `other`<br/>or<br/>`nice` | this is the default policy for all processes under Linux. It provides dynamic priorities based on the `nice` level of each process. Check below for setting this `nice` level for netdata. |
205
-| `batch` | This policy is similar to `other` in that it schedules the thread according to its dynamic priority (based on the `nice` value). The difference is that this policy will cause the scheduler to always assume that the thread is CPU-intensive. Consequently, the scheduler will apply a small scheduling penalty with respect to wake-up behavior, so that this thread is mildly disfavored in scheduling decisions. |
206
-| `fifo` | `fifo` can be used only with static priorities higher than 0, which means that when a `fifo` threads becomes runnable, it will always immediately preempt any currently running `other`, `batch`, or `idle` thread. `fifo` is a simple scheduling algorithm without time slicing. |
207
-| `rr` | a simple enhancement of `fifo`. Everything described above for `fifo` also applies to `rr`, except that each thread is allowed to run only for a maximum time quantum. |
205
+| `batch` | This policy is similar to `other` in that it schedules the thread according to its dynamic priority (based on the `nice` value). The difference is that this policy will cause the scheduler to always assume that the thread is CPU-intensive. Consequently, the scheduler will apply a small scheduling penalty with respect to wake-up behavior, so that this thread is mildly disfavored in scheduling decisions. |
206
+| `fifo` | `fifo` can be used only with static priorities higher than 0, which means that when a `fifo` threads becomes runnable, it will always immediately preempt any currently running `other`, `batch`, or `idle` thread. `fifo` is a simple scheduling algorithm without time slicing. |
207
+| `rr` | a simple enhancement of `fifo`. Everything described above for `fifo` also applies to `rr`, except that each thread is allowed to run only for a maximum time quantum. |
208
| `keep`<br/>or<br/>`none` | do not set scheduling policy, priority or nice level - i.e. keep running with whatever it is set already (e.g. by systemd). |
209
210
For more information see `man sched`.
@@ -278,11 +278,7 @@ all programs), edit `netdata.conf` and set:
278
process nice level = -1
279
```
280
281
-then execute this to [restart Netdata](/packaging/installer/README.md#maintaining-a-netdata-agent-installation):
282
-
283
-```sh
284
-sudo systemctl restart netdata
285
-```
281
+then [restart Netdata](/docs/netdata-agent/start-stop-restart.md):
282
283
#### Example 2: Netdata with nice -1 on systemd systems
284
@@ -332,7 +328,7 @@ will roughly get the number of threads running.
328
The system does this for speed. Having a separate memory arena for each thread, allows the threads to run in parallel in
329
multi-core systems, without any locks between them.
330
335
-This behaviour is system specific. For example, the chart above when running
331
+This behavior is system specific. For example, the chart above when running
332
Netdata on Alpine Linux (that uses **musl** instead of **glibc**) is this:
333
334
@@ -364,9 +360,9 @@ accounts the whole pages, even if parts of them are actually used).
360
361
When you compile Netdata with debugging:
362
367
-1. compiler optimizations for your CPU are disabled (Netdata will run somewhat slower)
363
+1. compiler optimizations for your CPU are disabled (Netdata will run somewhat slower)
364
369
-2. a lot of code is added all over netdata, to log debug messages to `/var/log/netdata/debug.log`. However, nothing is
365
+2. a lot of code is added all over netdata, to log debug messages to `/var/log/netdata/debug.log`. However, nothing is
366
printed by default. Netdata allows you to select which sections of Netdata you want to trace. Tracing is activated
367
via the config option `debug flags`. It accepts a hex number, to enable or disable specific sections. You can find
368
the options supported at [log.h](https://raw.githubusercontent.com/netdata/netdata/master/src/libnetdata/log/log.h).
@@ -404,9 +400,9 @@ To provide stack traces, **you need to have Netdata compiled with debugging**. T
400
401
Then you need to be in one of the following 2 cases:
402
407
-1. Netdata crashes and you have a core dump
403
+1. Netdata crashes and you have a core dump
404
409
-2. you can reproduce the crash
405
+2. you can reproduce the crash
406
407
If you are not on these cases, you need to find a way to be (i.e. if your system does not produce core dumps, check your
408
distro documentation to enable them).
src/exporting/README.md
+40
-52
@@ -1,13 +1,3 @@
1
-<!--
2
-title: "Exporting reference"
3
-description: "With the exporting engine, you can archive your Netdata metrics to multiple external databases for long-term storage or further analysis."
4
-sidebar_label: "Export"
5
-custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/exporting/README.md"
6
-learn_status: "Published"
7
-learn_rel_path: "Integrations/Export"
8
-learn_doc_purpose: "Explain the exporting engine options and all of our the exporting connectors options"
9
--->
10
-
1
# Exporting reference
2
3
Welcome to the exporting engine reference guide. This guide contains comprehensive information about enabling,
@@ -18,7 +8,7 @@ For a quick introduction to the exporting engine's features, read our doc on [ex
8
databases](/docs/exporting-metrics/README.md), or jump in to [enabling a connector](/docs/exporting-metrics/enable-an-exporting-connector.md).
9
10
The exporting engine has a modular structure and supports metric exporting via multiple exporting connector instances at
21
-the same time. You can have different update intervals and filters configured for every exporting connector instance.
11
+the same time. You can have different update intervals and filters configured for every exporting connector instance.
12
13
When you enable the exporting engine and a connector, the Netdata Agent exports metrics _beginning from the time you
14
restart its process_, not the entire [database of long-term metrics](/docs/netdata-agent/configuration/optimizing-metrics-database/change-metrics-storage.md).
@@ -37,24 +27,24 @@ The exporting engine uses a number of connectors to send Netdata metrics to exte
27
[list of supported databases](/docs/exporting-metrics/README.md#supported-databases) for information on which
28
connector to enable and configure for your database of choice.
29
40
-- [**AWS Kinesis Data Streams**](/src/exporting/aws_kinesis/README.md): Metrics are sent to the service in `JSON`
30
+- [**AWS Kinesis Data Streams**](/src/exporting/aws_kinesis/README.md): Metrics are sent to the service in `JSON`
31
format.
42
-- [**Google Cloud Pub/Sub Service**](/src/exporting/pubsub/README.md): Metrics are sent to the service in `JSON`
32
+- [**Google Cloud Pub/Sub Service**](/src/exporting/pubsub/README.md): Metrics are sent to the service in `JSON`
33
format.
44
-- [**Graphite**](/src/exporting/graphite/README.md): A plaintext interface. Metrics are sent to the database server as
34
+- [**Graphite**](/src/exporting/graphite/README.md): A plaintext interface. Metrics are sent to the database server as
35
`prefix.hostname.chart.dimension`. `prefix` is configured below, `hostname` is the hostname of the machine (can
36
also be configured). Learn more in our guide to [export and visualize Netdata metrics in
37
Graphite](/src/exporting/graphite/README.md).
48
-- [**JSON** document databases](/src/exporting/json/README.md)
49
-- [**OpenTSDB**](/src/exporting/opentsdb/README.md): Use a plaintext or HTTP interfaces. Metrics are sent to
38
+- [**JSON** document databases](/src/exporting/json/README.md)
39
+- [**OpenTSDB**](/src/exporting/opentsdb/README.md): Use a plaintext or HTTP interfaces. Metrics are sent to
40
OpenTSDB as `prefix.chart.dimension` with tag `host=hostname`.
51
-- [**MongoDB**](/src/exporting/mongodb/README.md): Metrics are sent to the database in `JSON` format.
52
-- [**Prometheus**](/src/exporting/prometheus/README.md): Use an existing Prometheus installation to scrape metrics
41
+- [**MongoDB**](/src/exporting/mongodb/README.md): Metrics are sent to the database in `JSON` format.
42
+- [**Prometheus**](/src/exporting/prometheus/README.md): Use an existing Prometheus installation to scrape metrics
43
from node using the Netdata API.
54
-- [**Prometheus remote write**](/src/exporting/prometheus/remote_write/README.md). A binary snappy-compressed protocol
44
+- [**Prometheus remote write**](/src/exporting/prometheus/remote_write/README.md). A binary snappy-compressed protocol
45
buffer encoding over HTTP. Supports many [storage
46
providers](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage).
57
-- [**TimescaleDB**](/src/exporting/TIMESCALE.md): Use a community-built connector that takes JSON streams from a
47
+- [**TimescaleDB**](/src/exporting/TIMESCALE.md): Use a community-built connector that takes JSON streams from a
48
Netdata client and writes them to a TimescaleDB table.
49
50
### Chart filtering
@@ -77,17 +67,17 @@ http://localhost:19999/api/v1/allmetrics?format=shell&filter=system.*
67
68
Netdata supports three modes of operation for all exporting connectors:
69
80
-- `as-collected` sends to external databases the metrics as they are collected, in the units they are collected.
70
+- `as-collected` sends to external databases the metrics as they are collected, in the units they are collected.
71
So, counters are sent as counters and gauges are sent as gauges, much like all data collectors do. For example,
72
to calculate CPU utilization in this format, you need to know how to convert kernel ticks to percentage.
73
84
-- `average` sends to external databases normalized metrics from the Netdata database. In this mode, all metrics
74
+- `average` sends to external databases normalized metrics from the Netdata database. In this mode, all metrics
75
are sent as gauges, in the units Netdata uses. This abstracts data collection and simplifies visualization, but
76
you will not be able to copy and paste queries from other sources to convert units. For example, CPU utilization
77
percentage is calculated by Netdata, so Netdata will convert ticks to percentage and send the average percentage
78
to the external database.
79
90
-- `sum` or `volume`: the sum of the interpolated values shown on the Netdata graphs is sent to the external
80
+- `sum` or `volume`: the sum of the interpolated values shown on the Netdata graphs is sent to the external
81
database. So, if Netdata is configured to send data to the database every 10 seconds, the sum of the 10 values
82
shown on the Netdata charts will be used.
83
@@ -102,7 +92,7 @@ see in Netdata, which is not necessarily true for the other modes of operation.
92
93
### Independent operation
94
105
-This code is smart enough, not to slow down Netdata, independently of the speed of the external database server.
95
+This code is smart enough, not to slow down Netdata, independently of the speed of the external database server.
96
97
> ❗ You should keep in mind though that many exporting connector instances can consume a lot of CPU resources if they
98
> run their batches at the same time. You can set different update intervals for every exporting connector instance,
@@ -111,7 +101,7 @@ This code is smart enough, not to slow down Netdata, independently of the speed
101
## Configuration
102
103
Here are the configuration blocks for every supported connector. Your current `exporting.conf` file may look a little
114
-different.
104
+different.
105
106
You can configure each connector individually using the available [options](#options). The
107
`[graphite:my_graphite_instance]` block contains examples of some of these additional options in action.
@@ -192,23 +182,23 @@ You can configure each connector individually using the available [options](#opt
182
183
### Sections
184
195
-- `[exporting:global]` is a section where you can set your defaults for all exporting connectors
196
-- `[prometheus:exporter]` defines settings for Prometheus exporter API queries (e.g.:
185
+- `[exporting:global]` is a section where you can set your defaults for all exporting connectors
186
+- `[prometheus:exporter]` defines settings for Prometheus exporter API queries (e.g.:
187
`http://NODE:19999/api/v1/allmetrics?format=prometheus&help=yes&source=as-collected`).
198
-- `[<type>:<name>]` keeps settings for a particular exporting connector instance, where:
199
- - `type` selects the exporting connector type: graphite | opentsdb:telnet | opentsdb:http |
188
+- `[<type>:<name>]` keeps settings for a particular exporting connector instance, where:
189
+- `type` selects the exporting connector type: graphite | opentsdb:telnet | opentsdb:http |
190
prometheus_remote_write | json | kinesis | pubsub | mongodb. For graphite, opentsdb,
191
json, and prometheus_remote_write connectors you can also use `:http` or `:https` modifiers
192
(e.g.: `opentsdb:https`).
203
- - `name` can be arbitrary instance name you chose.
193
+- `name` can be arbitrary instance name you chose.
194
195
### Options
196
197
Configure individual connectors and override any global settings with the following options.
198
209
-- `enabled = yes | no`, enables or disables an exporting connector instance
199
+- `enabled = yes | no`, enables or disables an exporting connector instance
200
211
-- `destination = host1 host2 host3 ...`, accepts **a space separated list** of hostnames, IPs (IPv4 and IPv6) and
201
+- `destination = host1 host2 host3 ...`, accepts **a space separated list** of hostnames, IPs (IPv4 and IPv6) and
202
ports to connect to. Netdata will use the **first available** to send the metrics.
203
204
The format of each item in this list, is: `[PROTOCOL:]IP[:PORT]`.
@@ -246,48 +236,48 @@ Configure individual connectors and override any global settings with the follow
236
237
For the Pub/Sub exporting connector `destination` can be set to a specific service endpoint.
238
249
-- `data source = as collected`, or `data source = average`, or `data source = sum`, selects the kind of data that will
239
+- `data source = as collected`, or `data source = average`, or `data source = sum`, selects the kind of data that will
240
be sent to the external database.
241
252
-- `hostname = my-name`, is the hostname to be used for sending data to the external database server. By default this
242
+- `hostname = my-name`, is the hostname to be used for sending data to the external database server. By default this
243
is `[global].hostname`.
244
255
-- `prefix = Netdata`, is the prefix to add to all metrics.
245
+- `prefix = Netdata`, is the prefix to add to all metrics.
246
257
-- `update every = 10`, is the number of seconds between sending data to the external database. Netdata will add some
247
+- `update every = 10`, is the number of seconds between sending data to the external database. Netdata will add some
248
randomness to this number, to prevent stressing the external server when many Netdata servers send data to the same
249
database. This randomness does not affect the quality of the data, only the time they are sent.
250
261
-- `buffer on failures = 10`, is the number of iterations (each iteration is `update every` seconds) to buffer data,
251
+- `buffer on failures = 10`, is the number of iterations (each iteration is `update every` seconds) to buffer data,
252
when the external database server is not available. If the server fails to receive the data after that many
253
failures, data loss on the connector instance is expected (Netdata will also log it).
254
265
-- `timeout ms = 20000`, is the timeout in milliseconds to wait for the external database server to process the data.
255
+- `timeout ms = 20000`, is the timeout in milliseconds to wait for the external database server to process the data.
256
By default this is `2 * update_every * 1000`.
257
268
-- `send hosts matching = localhost *` includes one or more space separated patterns, using `*` as wildcard (any number
258
+- `send hosts matching = localhost *` includes one or more space separated patterns, using `*` as wildcard (any number
259
of times within each pattern). The patterns are checked against the hostname (the localhost is always checked as
260
`localhost`), allowing us to filter which hosts will be sent to the external database when this Netdata is a central
261
Netdata aggregating multiple hosts. A pattern starting with `!` gives a negative match. So to match all hosts named
262
`*db*` except hosts containing `*child*`, use `!*child* *db*` (so, the order is important: the first
263
pattern matching the hostname will be used - positive or negative).
264
275
-- `send charts matching = *` includes one or more space separated patterns, using `*` as wildcard (any number of times
265
+- `send charts matching = *` includes one or more space separated patterns, using `*` as wildcard (any number of times
266
within each pattern). The patterns are checked against both chart id and chart name. A pattern starting with `!`
267
gives a negative match. So to match all charts named `apps.*` except charts ending in `*reads`, use `!*reads
268
apps.*` (so, the order is important: the first pattern matching the chart id or the chart name will be used -
269
positive or negative). There is also a URL parameter `filter` that can be used while querying `allmetrics`. The URL
270
parameter has a higher priority than the configuration option.
271
282
-- `send names instead of ids = yes | no` controls the metric names Netdata should send to the external database.
272
+- `send names instead of ids = yes | no` controls the metric names Netdata should send to the external database.
273
Netdata supports names and IDs for charts and dimensions. Usually IDs are unique identifiers as read by the system
274
and names are human friendly labels (also unique). Most charts and metrics have the same ID and name, but in several
275
cases they are different: disks with device-mapper, interrupts, QoS classes, statsd synthetic charts, etc.
276
287
-- `send configured labels = yes | no` controls if host labels defined in the `[host labels]` section in `netdata.conf`
277
+- `send configured labels = yes | no` controls if host labels defined in the `[host labels]` section in `netdata.conf`
278
should be sent to the external database
279
290
-- `send automatic labels = yes | no` controls if automatically created labels, like `_os_name` or `_architecture`
280
+- `send automatic labels = yes | no` controls if automatically created labels, like `_os_name` or `_architecture`
281
should be sent to the external database
282
283
## HTTPS
@@ -302,14 +292,14 @@ HTTPS communication between Netdata and an external database. You can set up a r
292
Netdata creates five charts in the dashboard, under the **Netdata Monitoring** section, to help you monitor the health
293
and performance of the exporting engine itself:
294
305
-1. **Buffered metrics**, the number of metrics Netdata added to the buffer for dispatching them to the
295
+1. **Buffered metrics**, the number of metrics Netdata added to the buffer for dispatching them to the
296
external database server.
297
308
-2. **Exporting data size**, the amount of data (in KB) Netdata added the buffer.
298
+2. **Exporting data size**, the amount of data (in KB) Netdata added the buffer.
299
310
-3. **Exporting operations**, the number of operations performed by Netdata.
300
+3. **Exporting operations**, the number of operations performed by Netdata.
301
312
-4. **Exporting thread CPU usage**, the CPU resources consumed by the Netdata thread, that is responsible for sending
302
+4. **Exporting thread CPU usage**, the CPU resources consumed by the Netdata thread, that is responsible for sending
303
the metrics to the external database server.
304
305
@@ -318,10 +308,8 @@ and performance of the exporting engine itself:
308
309
Netdata adds 3 alerts:
310
321
-1. `exporting_last_buffering`, number of seconds since the last successful buffering of exported data
322
-2. `exporting_metrics_sent`, percentage of metrics sent to the external database server
323
-3. `exporting_metrics_lost`, number of metrics lost due to repeating failures to contact the external database server
311
+1. `exporting_last_buffering`, number of seconds since the last successful buffering of exported data
312
+2. `exporting_metrics_sent`, percentage of metrics sent to the external database server
313
+3. `exporting_metrics_lost`, number of metrics lost due to repeating failures to contact the external database server
314
315
326
-
327
-
src/exporting/TIMESCALE.md
+4
-11
@@ -1,12 +1,3 @@
1
-<!--
2
-title: "Writing metrics to TimescaleDB"
3
-description: "Send Netdata metrics to TimescaleDB for long-term archiving and further analysis."
4
-custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/exporting/TIMESCALE.md"
5
-sidebar_label: "Writing metrics to TimescaleDB"
6
-learn_status: "Published"
7
-learn_rel_path: "Integrations/Export"
8
--->
9
-
1
# Writing metrics to TimescaleDB
2
3
Thanks to Netdata's community of developers and system administrators, and Mahlon Smith
@@ -23,14 +14,18 @@ What's TimescaleDB? Here's how their team defines the project on their [GitHub p
14
To get started archiving metrics to TimescaleDB right away, check out Mahlon's [`netdata-timescale-relay`
15
repository](https://github.com/mahlonsmith/netdata-timescale-relay) on GitHub. Please be aware that backends subsystem
16
was removed and Netdata configuration should be moved to the new `exporting.conf` configuration file. Use
17
+
18
```conf
19
[json:my_instance]
20
```
21
+
22
in `exporting.conf` instead of
23
+
24
```conf
25
[backend]
26
type = json
27
```
28
+
29
in `netdata.conf`.
30
31
This small program takes JSON streams from a Netdata client and writes them to a PostgreSQL (aka TimescaleDB) table.
@@ -67,5 +62,3 @@ blog](https://blog.timescale.com/blog/writing-it-metrics-from-netdata-to-timesca
62
63
Thank you to Mahlon, Rune, TimescaleDB, and the members of the Netdata community that requested and then built this
64
exporting connection between Netdata and TimescaleDB!
70
-
71
-
src/exporting/WALKTHROUGH.md
+4
-6
@@ -37,7 +37,7 @@ This stack will offer you visibility into your application and systems performan
37
To begin let's create our container which we will install Netdata on. We need to run a container, forward the necessary
38
port that Netdata listens on, and attach a tty so we can interact with the bash shell on the container. But before we do
39
this we want name resolution between the two containers to work. In order to accomplish this we will create a
40
-user-defined network and attach both containers to this network. The first command we should run is:
40
+user-defined network and attach both containers to this network. The first command we should run is:
41
42
```sh
43
docker network create --driver bridge netdata-tutorial
@@ -90,15 +90,15 @@ We will be installing Prometheus in a container for purpose of demonstration. Wh
90
container I would like to walk through the install process and setup on a fresh container. This will allow anyone
91
reading to migrate this tutorial to a VM or Server of any sort.
92
93
-Let's start another container in the same fashion as we did the Netdata container.
93
+Let's start another container in the same fashion as we did the Netdata container.
94
95
```sh
96
docker run -it --name prometheus --hostname prometheus \
97
--network=netdata-tutorial -p 9090:9090 centos:latest '/bin/bash'
98
-```
98
+```
99
100
This should drop you into a shell once again. Once there quickly install your favorite editor as we will be editing
101
-files later in this tutorial.
101
+files later in this tutorial.
102
103
```sh
104
yum install vim -y
@@ -256,5 +256,3 @@ deployments automatically register Netdata services into Consul and Prometheus a
256
achieved you do not have to think about the monitoring system until Prometheus cannot keep up with your scale. Once this
257
happens there are options presented in the Prometheus documentation for solving this. Hope this was helpful, happy
258
monitoring.
259
-
260
-
src/go/plugin/go.d/README.md
-11
@@ -1,14 +1,3 @@
1
-<!--
2
-title: go.d.plugin
3
-description: "go.d.plugin is an external plugin for Netdata, responsible for running individual data collectors written in Go."
4
-custom_edit_url: "/src/go/plugin/go.d/README.md"
5
-sidebar_label: "go.d.plugin"
6
-learn_status: "Published"
7
-learn_topic_type: "Tasks"
8
-learn_rel_path: "Developers/External plugins/go.d.plugin"
9
-sidebar_position: 1
10
--->
11
-
1
# go.d.plugin
2
3
`go.d.plugin` is a [Netdata](https://github.com/netdata/netdata) external plugin. It is an **orchestrator** for data
src/go/plugin/go.d/docs/how-to-write-a-module.md
+17
-29
@@ -1,14 +1,3 @@
1
-<!--
2
-title: "How to write a Netdata collector in Go"
3
-description: "This guide will walk you through the technical implementation of writing a new Netdata collector in Golang, with tips on interfaces, structure, configuration files, and more."
4
-custom_edit_url: "/src/go/plugin/go.d/docs/how-to-write-a-module.md"
5
-sidebar_label: "How to write a Netdata collector in Go"
6
-learn_status: "Published"
7
-learn_topic_type: "Tasks"
8
-learn_rel_path: "Developers/External plugins/go.d.plugin"
9
-sidebar_position: 20
10
--->
11
-
1
# How to write a Netdata collector in Go
2
3
## Prerequisites
@@ -22,7 +11,7 @@ sidebar_position: 20
11
12
## Write and test a simple collector
13
25
-> :exclamation: You can skip most of these steps if you first experiment directy with the existing
14
+> :exclamation: You can skip most of these steps if you first experiment directly with the existing
15
> [example module](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/modules/example), which
16
> will
17
> give you an idea of how things work.
@@ -33,9 +22,9 @@ The steps are:
22
23
- Add the source code
24
to [`modules/example2/`](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/modules).
36
- - [module interface](#module-interface).
37
- - [suggested module layout](#module-layout).
38
- - [helper packages](#helper-packages).
25
+ - [module interface](#module-interface).
26
+ - [suggested module layout](#module-layout).
27
+ - [helper packages](#helper-packages).
28
- Add the configuration
29
to [`config/go.d/example2.conf`](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/config/go.d).
30
- Add the module
@@ -58,7 +47,7 @@ The steps are:
47
48
Every module should implement the following interface:
49
61
-```
50
+```go
51
type Module interface {
52
Init() bool
53
Check() bool
@@ -75,7 +64,7 @@ type Module interface {
64
65
We propose to use the following template:
66
78
-```
67
+```go
68
// example.go
69
70
func (e *Example) Init() bool {
@@ -97,7 +86,7 @@ func (e *Example) Init() bool {
86
}
87
```
88
100
-Move specific initialization methods into the `init.go` file. See [suggested module layout](#module-Layout).
89
+Move specific initialization methods into the `init.go` file. See [suggested module layout](#module-layout).
90
91
### Check method
92
@@ -108,7 +97,7 @@ Move specific initialization methods into the `init.go` file. See [suggested mod
97
The simplest way to implement `Check` is to see if we are getting any metrics from `Collect`. A lot of modules use such
98
approach.
99
111
-```
100
+```go
101
// example.go
102
103
func (e *Example) Check() bool {
@@ -134,7 +123,7 @@ it contains charts and dimensions structs.
123
124
Usually charts initialized in `Init` and `Chart` method just returns the charts instance:
125
137
-```
126
+```go
127
// example.go
128
129
func (e *Example) Charts() *Charts {
@@ -151,7 +140,7 @@ func (e *Example) Charts() *Charts {
140
141
We propose to use the following template:
142
154
-```
143
+```go
144
// example.go
145
146
func (e *Example) Collect() map[string]int64 {
@@ -167,7 +156,7 @@ func (e *Example) Collect() map[string]int64 {
156
}
157
```
158
170
-Move metrics collection logic into the `collect.go` file. See [suggested module layout](#module-Layout).
159
+Move metrics collection logic into the `collect.go` file. See [suggested module layout](#module-layout).
160
161
### Cleanup method
162
@@ -176,7 +165,7 @@ Move metrics collection logic into the `collect.go` file. See [suggested module
165
166
If you have nothing to clean up:
167
179
-```
168
+```go
169
// example.go
170
171
func (Example) Cleanup() {}
@@ -229,7 +218,7 @@ All the module initialization details should go in this file.
218
- make a function for each value that needs to be initialized.
219
- a function should return a value(s), not implicitly set/change any values in the main struct.
220
232
-```
221
+```go
222
// init.go
223
224
// Prefer this approach.
@@ -244,7 +233,7 @@ func (e *Example) initSomeValue() error {
233
m.someValue = someValue
234
return nil
235
}
247
-```
236
+```
237
238
### File `collect.go`
239
@@ -257,7 +246,7 @@ Feel free to split it into several files if you think it makes the code more rea
246
247
Use `collect_` prefix for the filenames: `collect_this.go`, `collect_that.go`, etc.
248
260
-```
249
+```go
250
// collect.go
251
252
func (e *Example) collect() (map[string]int64, error) {
@@ -273,10 +262,10 @@ func (e *Example) collect() (map[string]int64, error) {
262
263
> :exclamation: See the
264
> example: [`example_test.go`](https://github.com/netdata/netdata/blob/master/src/go/plugin/go.d/modules/example/example_test.go).
276
-
265
+>
266
> if you have no experience in testing we recommend starting
267
> with [testing package documentation](https://golang.org/pkg/testing/).
279
-
268
+>
269
> we use `assert` and `require` packages from [github.com/stretchr/testify](https://github.com/stretchr/testify)
270
> library,
271
> check [their documentation](https://pkg.go.dev/github.com/stretchr/testify).
@@ -299,4 +288,3 @@ be [`testdata`](https://golang.org/cmd/go/#hdr-Package_lists_and_patterns).
288
289
There are [some helper packages](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/pkg) for
290
writing a module.
302
-
src/health/README.md
+4
-2
@@ -2,9 +2,11 @@
2
3
Netdata offers two ways to receive alert notifications on external integrations. These methods work independently, which means you can enable both at the same time to send alert notifications to any number of endpoints.
4
5
-Both methods use a node's health alerts to generate the content of a notification.
5
+Both methods use a node's health alerts to generate the content of a notification.
6
7
-Read our documentation on [configuring alerts](/src/health/REFERENCE.md) to change the preconfigured thresholds or to create tailored alerts for your infrastructure.
7
+Read our documentation on [configuring alerts](/src/health/REFERENCE.md) to change the pre-configured thresholds or to create tailored alerts for your infrastructure.
8
+
9
+<!-- virtual links below, should not lead anywhere outside of the rendered Learn doc -->
10
11
- Netdata Cloud provides centralized alert notifications, utilizing the health status data already sent to Netdata Cloud from connected nodes to send alerts to configured integrations. [Supported integrations](/docs/alerts-&-notifications/notifications/centralized-cloud-notifications) include Amazon SNS, Discord, Slack, Splunk, and others.
12
src/health/REFERENCE.md
+5
-5
@@ -640,7 +640,7 @@ See our [simple patterns docs](/src/libnetdata/simple_pattern/README.md) for mor
640
Similar to host labels, the `chart labels` key can be used to filter if an alert will load or not for a specific chart, based on
641
whether these chart labels match or not.
642
643
-The list of chart labels present on each chart can be obtained from http://localhost:19999/api/v1/charts?all
643
+The list of chart labels present on each chart can be obtained from <http://localhost:19999/api/v1/charts?all>
644
645
For example, each `disk_space` chart defines a chart label called `mount_point` with each instance of this chart having
646
a value there of which mount point it monitors.
@@ -808,14 +808,14 @@ You can find all the variables that can be used for a given chart, using
808
Agent dashboard. For example, [variables for the `system.cpu` chart of the
809
registry](https://registry.my-netdata.io/api/v1/alarm_variables?chart=system.cpu).
810
811
-> If you don't know how to find the CHART_NAME, you can read about it [here](/src/web/README.md#charts).
811
+<!-- > If you don't know how to find the CHART_NAME, you can read about it [here](/src/web/README.md#charts). -->
812
813
Netdata supports 3 internal indexes for variables that will be used in health monitoring.
814
815
<details><summary>The variables below can be used in both chart alerts and context templates.</summary>
816
817
Although the `alarm_variables` link shows you variables for a particular chart, the same variables can also be used in
818
-templates for charts belonging to a given [context](/src/web/README.md#contexts). The reason is that all charts of a given
818
+templates for charts belonging to a given context. The reason is that all charts of a given
819
context are essentially identical, with the only difference being the family that identifies a particular hardware or software instance.
820
821
</details>
@@ -1064,7 +1064,7 @@ template: ml_5min_cpu_chart
1064
info: rolling 5min anomaly rate for system.cpu chart
1065
```
1066
1067
-The `lookup` line will calculate the average anomaly rate across all `system.cpu` dimensions over the last 5 minues. In this case
1067
+The `lookup` line will calculate the average anomaly rate across all `system.cpu` dimensions over the last 5 minutes. In this case
1068
Netdata will create one alert for the chart.
1069
1070
### Example 7 - [Anomaly rate](/src/ml/README.md#anomaly-rate) based node level alert
@@ -1083,7 +1083,7 @@ template: ml_5min_node
1083
info: rolling 5min anomaly rate for all ML enabled dims
1084
```
1085
1086
-The `lookup` line will use the `anomaly_rate` dimension of the `anomaly_detection.anomaly_rate` ML chart to calculate the average [node level anomaly rate](/src/ml/README.md#node-anomaly-rate) over the last 5 minutes.
1086
+The `lookup` line will use the `anomaly_rate` dimension of the `anomaly_detection.anomaly_rate` ML chart to calculate the average [node level anomaly rate](/src/ml/README.md#anomaly-rate) over the last 5 minutes.
1087
1088
## Troubleshooting
1089
src/libnetdata/README.md
-11
@@ -1,14 +1,3 @@
1
-<!--
2
-title: "libnetdata"
3
-custom_edit_url: https://github.com/netdata/netdata/edit/master/src/libnetdata/README.md
4
-sidebar_label: "libnetdata"
5
-learn_status: "Published"
6
-learn_topic_type: "Tasks"
7
-learn_rel_path: "Developers/libnetdata"
8
--->
9
-
1
# libnetdata
2
3
`libnetdata` is a collection of library code that is used by all Netdata `C` programs.
13
-
14
-
src/registry/README.md
+27
-39
@@ -1,12 +1,3 @@
1
-<!--
2
-title: "Registry"
3
-description: "Netdata utilizes a central registry of machines/person GUIDs, URLs, and opt-in account information to provide unified cross-server dashboards."
4
-custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/registry/README.md"
5
-sidebar_label: "Registry"
6
-learn_status: "Published"
7
-learn_rel_path: "Configuration"
8
--->
9
-
1
# Registry
2
3
Netdata provides distributed monitoring.
@@ -14,21 +5,21 @@ Netdata provides distributed monitoring.
5
Traditional monitoring solutions centralize all the data to provide unified dashboards across all servers. Before
6
Netdata, this was the standard practice. However it has a few issues:
7
17
-1. due to the resources required, the number of metrics collected is limited.
18
-2. for the same reason, the data collection frequency is not that high, at best it will be once every 10 or 15 seconds,
8
+1. due to the resources required, the number of metrics collected is limited.
9
+2. for the same reason, the data collection frequency is not that high, at best it will be once every 10 or 15 seconds,
10
at worst every 5 or 10 mins.
20
-3. the central monitoring solution needs dedicated resources, thus becoming "another bottleneck" in the whole
11
+3. the central monitoring solution needs dedicated resources, thus becoming "another bottleneck" in the whole
12
ecosystem. It also requires maintenance, administration, etc.
22
-4. most centralized monitoring solutions are usually only good for presenting _statistics of past performance_ (i.e.
13
+4. most centralized monitoring solutions are usually only good for presenting _statistics of past performance_ (i.e.
14
cannot be used for real-time performance troubleshooting).
15
16
Netdata follows a different approach:
17
27
-1. data collection happens per second
28
-2. thousands of metrics per server are collected
29
-3. data do not leave the server where they are collected
30
-4. Netdata servers do not talk to each other
31
-5. your browser connects all the Netdata servers
18
+1. data collection happens per second
19
+2. thousands of metrics per server are collected
20
+3. data do not leave the server where they are collected
21
+4. Netdata servers do not talk to each other
22
+5. your browser connects all the Netdata servers
23
24
Using Netdata, your monitoring infrastructure is embedded on each server, limiting significantly the need of additional
25
resources. Netdata is blazingly fast, very resource efficient and utilizes server resources that already exist and are
@@ -46,31 +37,30 @@ etc.) are propagated to the new server, so that the new dashboard will come with
37
38
The registry keeps track of 4 entities:
39
49
-1. **machines**: i.e. the Netdata installations (a random GUID generated by each Netdata the first time it starts; we
40
+1. **machines**: i.e. the Netdata installations (a random GUID generated by each Netdata the first time it starts; we
41
call this **machine_guid**)
42
52
- For each Netdata installation (each `machine_guid`) the registry keeps track of the different URLs it has accessed.
43
+ For each Netdata installation (each `machine_guid`) the registry keeps track of the different URLs it has accessed.
44
54
-2. **persons**: i.e. the web browsers accessing the Netdata installations (a random GUID generated by the registry the
45
+2. **persons**: i.e. the web browsers accessing the Netdata installations (a random GUID generated by the registry the
46
first time it sees a new web browser; we call this **person_guid**)
47
57
- For each person, the registry keeps track of the Netdata installations it has accessed and their URLs.
48
+ For each person, the registry keeps track of the Netdata installations it has accessed and their URLs.
49
59
-3. **URLs** of Netdata installations (as seen by the web browsers)
50
+3. **URLs** of Netdata installations (as seen by the web browsers)
51
61
- For each URL, the registry keeps the URL and nothing more. Each URL is linked to _persons_ and _machines_. The only
52
+ For each URL, the registry keeps the URL and nothing more. Each URL is linked to _persons_ and _machines_. The only
53
way to find a URL is to know its **machine_guid** or have a **person_guid** it is linked to it.
54
64
-4. **accounts**: i.e. the information used to sign-in via one of the available sign-in methods. Depending on the
65
- method, this may include an email, or an email and a profile picture or avatar.
55
+4. **accounts**: i.e. the information used to sign-in via one of the available sign-in methods. Depending on the method, this may include an email, or an email and a profile picture or avatar.
56
57
For _persons_/_accounts_ and _machines_, the registry keeps links to _URLs_, each link with 2 timestamps (first time
58
seen, last time seen) and a counter (number of times it has been seen). *machines_, _persons_ and timestamps are stored
69
-in the Netdata registry regardless of whether you sign in or not.
59
+in the Netdata registry regardless of whether you sign in or not.
60
61
## Who talks to the registry?
62
73
-Your web browser **only**! If sending this information is against your policies, you
63
+Your web browser **only**! If sending this information is against your policies, you
64
can [run your own registry](#run-your-own-registry)
65
66
Your Netdata servers do not talk to the registry. This is a UML diagram of its operation:
@@ -158,9 +148,10 @@ pattern matching can be controlled with the following setting:
148
```
149
150
The settings are:
161
-- `yes` allows the pattern to match DNS names.
162
-- `no` disables DNS matching for the patterns (they only match IP addresses).
163
-- `heuristic` will estimate if the patterns should match FQDNs by the presence or absence of `:`s or alpha-characters.
151
+
152
+- `yes` allows the pattern to match DNS names.
153
+- `no` disables DNS matching for the patterns (they only match IP addresses).
154
+- `heuristic` will estimate if the patterns should match FQDNs by the presence or absence of `:`s or alpha-characters.
155
156
### Where is the registry database stored?
157
@@ -168,14 +159,13 @@ The settings are:
159
160
There can be up to 2 files:
161
171
-- `registry-log.db`, the transaction log
172
-
173
- all incoming requests that affect the registry are saved in this file in real-time.
162
+- `registry-log.db`, the transaction log
163
175
-- `registry.db`, the database
164
+ all incoming requests that affect the registry are saved in this file in real-time.
165
177
- every `[registry].registry save db every new entries` entries in `registry-log.db`, Netdata will save its database
178
- to `registry.db` and empty `registry-log.db`.
166
+- `registry.db`, the database
167
+
168
+ every `[registry].registry save db every new entries` entries in `registry-log.db`, Netdata will save its database to `registry.db` and empty `registry-log.db`.
169
170
Both files are machine readable text files.
171
@@ -213,5 +203,3 @@ ERROR 409: Cannot ACCESS netdata registry: https://registry.my-netdata.io respon
203
```
204
205
This error is printed on your web browser console (press F12 on your browser to see it).
216
-
217
-