@cryptotaxi247 / netdata-1 / commits / 9bdc1f595

Collecting metrics docs grammar pass (#17736)

* grammar pass * grammar pass * grammar pass and some edits * grammar pass * grammar pass on the whole dir and remove duplicates * simplify wording application-metrics.md * Update docs/collecting-metrics/application-metrics.md --------- Co-authored-by: ilyam8 <ilya@netdata.cloud>

Fotis Voutsas committed May 27, 2024 at 11:16 UTC 9bdc1f595e44fa4ceae70037b1eb6c2b3bebb4d5
7 files changed +129 -313
docs/collecting-metrics/application-metrics.md
+35 -83
@@ -1,83 +1,35 @@
1 -<!--
2 -title: "Collect application metrics with Netdata"
3 -sidebar_label: "Application metrics"
4 -description: "Monitor and troubleshoot every application on your infrastructure with per-second metrics, zero configuration, and meaningful charts."
5 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/collecting-metrics/application-metrics.md"
6 -learn_status: "Published"
7 -learn_topic_type: "Concepts"
8 -learn_rel_path: "Concepts"
9 --->
10 -
11 -# Collect application metrics with Netdata
12 -
13 -Netdata instantly collects per-second metrics from many different types of applications running on your systems, such as
14 -web servers, databases, message brokers, email servers, search platforms, and much more. Metrics collectors are
15 -pre-installed with every Netdata Agent and usually require zero configuration. Netdata also collects and visualizes
16 -resource utilization per application on Linux systems using `apps.plugin`.
17 -
18 -[**apps.plugin**](/src/collectors/apps.plugin/README.md) looks at the Linux process tree every second, much like `top` or
19 -`ps fax`, and collects resource utilization information on every running process. By reading the process tree, Netdata
20 -shows CPU, disk, networking, processes, and eBPF for every application or Linux user. Unlike `top` or `ps fax`, Netdata
21 -adds a layer of meaningful visualization on top of the process tree metrics, such as grouping applications into useful
22 -dimensions, and then creates per-application charts under the **Applications** section of a Netdata dashboard, per-user
23 -charts under **Users**, and per-user group charts under **User Groups**.
24 -
25 -Our most popular application collectors:
26 -
27 -- [Prometheus endpoints](/src/go/collectors/go.d.plugin/modules/prometheus/README.md): Gathers
28 - metrics from one or more Prometheus endpoints that use the OpenMetrics exposition format. Auto-detects more than 600
29 - endpoints.
30 -- [Web server logs (Apache, NGINX)](/src/go/collectors/go.d.plugin/modules/weblog/README.md):
31 - Tail access logs and provide very detailed web server performance statistics. This module is able to parse 200k+
32 - rows in less than half a second.
33 -- [MySQL](/src/go/collectors/go.d.plugin/modules/mysql/README.md): Collect database global,
34 - replication, and per-user statistics.
35 -- [Redis](/src/go/collectors/go.d.plugin/modules/redis/README.md): Monitor database status by
36 - reading the server's response to the `INFO` command.
37 -- [Apache](/src/go/collectors/go.d.plugin/modules/apache/README.md): Collect Apache web server
38 - performance metrics via the `server-status?auto` endpoint.
39 -- [Nginx](/src/go/collectors/go.d.plugin/modules/nginx/README.md): Monitor web server status
40 - information by gathering metrics via `ngx_http_stub_status_module`.
41 -- [Postgres](/src/go/collectors/go.d.plugin/modules/postgres/README.md): Collect database health
42 - and performance metrics.
43 -- [ElasticSearch](/src/go/collectors/go.d.plugin/modules/elasticsearch/README.md): Collect search
44 - engine performance and health statistics. Optionally collects per-index metrics.
45 -- [PHP-FPM](/src/go/collectors/go.d.plugin/modules/phpfpm/README.md): Collect application summary
46 - and processes health metrics by scraping the status page (`/status?full`).
47 -
48 -Our [supported collectors list](/src/collectors/COLLECTORS.md#service-and-application-collectors) shows all Netdata's
49 -application metrics collectors, including those for containers/k8s clusters.
50 -
51 -## Collect metrics from applications running on Windows
52 -
53 -Netdata is fully capable of collecting and visualizing metrics from applications running on Windows systems. The only
54 -caveat is that you must [install Netdata](/packaging/installer/README.md) on a separate system or a compatible VM because there
55 -is no native Windows version of the Netdata Agent.
56 -
57 -Once you have Netdata running on that separate system, you can follow the [collectors configuration reference](/src/collectors/REFERENCE.md) documentation to tell the collector to look for exposed metrics on the Windows system's IP
58 -address or hostname, plus the applicable port.
59 -
60 -For example, you have a MySQL database with a root password of `my-secret-pw` running on a Windows system with the IP
61 -address 203.0.113.0. you can configure the [MySQL
62 -collector](/src/go/collectors/go.d.plugin/modules/mysql/README.md) to look at `203.0.113.0:3306`:
63 -
64 -```yml
65 -jobs:
66 - - name: local
67 - dsn: root:my-secret-pw@tcp(203.0.113.0:3306)/
68 -```
69 -
70 -This same logic applies to any application in our [supported collectors
71 -list](/src/collectors/COLLECTORS.md#service-and-application-collectors) that can run on Windows.
72 -
73 -## What's next?
74 -
75 -If you haven't yet seen the [supported collectors list](/src/collectors/COLLECTORS.md) give it a once-over for any
76 -additional applications you may want to monitor using Netdata's native collectors, or the [generic Prometheus
77 -collector](/src/go/collectors/go.d.plugin/modules/prometheus/README.md).
78 -
79 -Collecting all the available metrics on your nodes, and across your entire infrastructure, is just one piece of the
80 -puzzle. Next, learn more about Netdata's famous real-time visualizations by [seeing an overview of your
81 -infrastructure](/docs/dashboards-and-charts/home-tab.md) using Netdata Cloud.
82 -
83 -
1 +# Collect Application Metrics with Netdata
2 +
3 +Netdata collects per-second metrics from a wide variety of applications running on your systems, including web servers,
4 +databases, message brokers, email servers, search platforms, and more. These metrics collectors are pre-installed with
5 +every Netdata Agent and typically require no configuration. Netdata also
6 +uses [`apps.plugin`](/src/collectors/apps.plugin/README.md) to gather and visualize resource utilization per application
7 +on Linux systems.
8 +
9 +The `apps.plugin` inspects the Linux process tree every second, similar to `top` or `ps fax`, and collects resource
10 +utilization data for every running process. However, Netdata goes a step further: instead of just displaying raw data,
11 +it transforms it into easy-to-understand charts. Rather than presenting a long list of processes, Netdata categorizes
12 +applications into meaningful groups, such as "web servers" or "databases." Each category has its own charts in the
13 +**Applications** section of your Netdata dashboard. Additionally, there are charts for individual users and user groups
14 +under the **Users** and **User Groups** sections.
15 +
16 +In addition to charts, `apps.plugin` offers the **Processes** [Function](/docs/top-monitoring-netdata-functions.md),
17 +which visualizes process entries in a table and allows for intuitive exploration of the processes. For more details on
18 +how the visualization of Functions works, check out the documentation on
19 +the [Top tab](/docs/dashboards-and-charts/top-tab.md).
20 +
21 +Popular application collectors:
22 +
23 +| Collector | Description |
24 +|--------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
25 +| [Prometheus](/src/go/collectors/go.d.plugin/modules/prometheus/README.md) | Gathers metrics from one or more Prometheus endpoints that use the OpenMetrics exposition format. Auto-detects more than 600 endpoints. |
26 +| [Web server logs (Apache, NGINX)](/src/go/collectors/go.d.plugin/modules/weblog/README.md) | Tails access logs and provides very detailed web server performance statistics. This module is able to parse 200k+ rows in less than half a second. |
27 +| [MySQL](/src/go/collectors/go.d.plugin/modules/mysql/README.md) | Collects database global, replication, and per-user statistics. |
28 +| [Redis](/src/go/collectors/go.d.plugin/modules/redis/README.md) | Monitors database status by reading the server's response to the `INFO` command. |
29 +| [Apache](/src/go/collectors/go.d.plugin/modules/apache/README.md) | Collects Apache web server performance metrics via the `server-status?auto` endpoint. |
30 +| [Nginx](/src/go/collectors/go.d.plugin/modules/nginx/README.md) | Monitors web server status information by gathering metrics via `ngx_http_stub_status_module`. |
31 +| [Postgres](/src/go/collectors/go.d.plugin/modules/postgres/README.md) | Collects database health and performance metrics. |
32 +| [ElasticSearch](/src/go/collectors/go.d.plugin/modules/elasticsearch/README.md) | Collects search engine performance and health statistics. Can optionally collect per-index metrics as well. |
33 +
34 +Check available [data collection integrations](/src/collectors/COLLECTORS.md#available-data-collection-integrations) for
35 +a comprehensive view to all the integrations you can use to gather metrics with Netdata.
docs/collecting-metrics/container-metrics.md
+21 -79
@@ -1,48 +1,20 @@
1 -<!--
2 -title: "Collect container metrics with Netdata"
3 -sidebar_label: "Container metrics"
4 -description: "Use Netdata to collect per-second utilization and application-level metrics from Linux/Docker containers and Kubernetes clusters."
5 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/collecting-metrics/container-metrics.md"
6 -learn_status: "Published"
7 -learn_topic_type: "Concepts"
8 -learn_rel_path: "Concepts"
9 --->
10 -
11 -# Collect container metrics with Netdata
1 +# Collect Container Metrics with Netdata
2
3 Thanks to close integration with Linux cgroups and the virtual files it maintains under `/sys/fs/cgroup`, Netdata can
14 -monitor the health, status, and resource utilization of many different types of Linux containers.
15 -
16 -Netdata uses [cgroups.plugin](/src/collectors/cgroups.plugin/README.md) to poll `/sys/fs/cgroup` and convert the raw data
17 -into human-readable metrics and meaningful visualizations. Through cgroups, Netdata is compatible with **all Linux
18 -containers**, such as Docker, LXC, LXD, Libvirt, systemd-nspawn, and more. Read more about [Docker-specific
19 -monitoring](#collect-docker-metrics) below.
20 -
21 -Netdata also has robust **Kubernetes monitoring** support thanks to a
22 -[Helmchart](/packaging/installer/methods/kubernetes.md) to automate deployment, collectors for k8s agent services, and
23 -robust [service discovery](https://github.com/netdata/agent-service-discovery/#service-discovery) to monitor the
24 -services running inside of pods in your k8s cluster. Read more about [Kubernetes
25 -monitoring](#collect-kubernetes-metrics) below.
26 -
27 -A handful of additional collectors gather metrics from container-related services, such as
28 -[dockerd](/src/go/collectors/go.d.plugin/modules/docker/README.md) or [Docker
29 -Engine](/src/go/collectors/go.d.plugin/modules/docker_engine/README.md). You can find all
30 -container collectors in our supported collectors list under the
31 -[containers/VMs](/src/collectors/COLLECTORS.md#containers-and-vms) and
32 -[Kubernetes](/src/collectors/COLLECTORS.md#containers-and-vms) headings.
4 +monitor health, status, and resource utilization of many different types of Linux containers.
5 +
6 +Netdata uses the [cgroups.plugin](/src/collectors/cgroups.plugin/README.md) to poll `/sys/fs/cgroup` and convert the raw data into human-readable metrics and meaningful visualizations. Through cgroups, Netdata is compatible with **all Linux containers**, such as Docker, LXC, LXD, Libvirt, systemd-nspawn, and more. Read more about [Docker-specific monitoring](#collect-docker-metrics) below.
7 +
8 +Netdata also has robust **Kubernetes monitoring** support thanks to a [Helmchart](/packaging/installer/methods/kubernetes.md) to automate deployment, collectors for k8s agent services, and robust [service discovery](https://github.com/netdata/agent-service-discovery/#service-discovery) to monitor the services running inside of pods in your k8s cluster. Read more about [Kubernetes monitoring](#collect-kubernetes-metrics) below.
9 +
10 +A handful of additional collectors gather metrics from container-related services, such as [dockerd](/src/go/collectors/go.d.plugin/modules/docker/README.md) or [Docker Engine](/src/go/collectors/go.d.plugin/modules/docker_engine/README.md). You can find all
11 +container collectors in our data collection integrations list under the [containers/VMs](/src/collectors/COLLECTORS.md#containers-and-vms) and [Kubernetes](/src/collectors/COLLECTORS.md#kubernetes) sections.
12
13 ## Collect Docker metrics
14
36 -Netdata has robust Docker monitoring thanks to the aforementioned
37 -[cgroups.plugin](/src/collectors/cgroups.plugin/README.md). By polling cgroups every second, Netdata can produce meaningful
38 -visualizations about the CPU, memory, disk, and network utilization of all running containers on the host system with
39 -zero configuration.
15 +Netdata has robust Docker monitoring thanks to the aforementioned [cgroups.plugin](/src/collectors/cgroups.plugin/README.md).
16
41 -Netdata also collects metrics from applications running inside of Docker containers. For example, if you create a MySQL
42 -database container using `docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag`, it exposes
43 -metrics on port 3306. You can configure the [MySQL
44 -collector](/src/go/collectors/go.d.plugin/modules/mysql/README.md) to look at `127.0.0.0:3306` for
45 -MySQL metrics:
17 +Netdata also collects metrics from applications running inside of Docker containers. For example, if you create a MySQL database container using `docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag`, it exposes metrics on port 3306. You can configure the [MySQL collector](/src/go/collectors/go.d.plugin/modules/mysql/README.md) to look at `127.0.0.0:3306` for MySQL metrics:
18
19 ```yml
20 jobs:
@@ -50,52 +22,22 @@ jobs:
22 dsn: root:my-secret-pw@tcp(127.0.0.1:3306)/
23 ```
24
53 -Netdata then collects metrics from the container itself, but also dozens [MySQL-specific
54 -metrics](/src/go/collectors/go.d.plugin/modules/mysql/README.md#charts) as well.
25 +Netdata then collects metrics from the container itself, but also dozens [MySQL-specific metrics](/src/go/collectors/go.d.plugin/modules/mysql/README.md#charts) as well.
26
27 ### Collect metrics from applications running in Docker containers
28
58 -You could use this technique to monitor an entire infrastructure of Docker containers. The same [enable and configure](/src/collectors/REFERENCE.md) procedures apply whether an application runs on the host system or inside
59 -a container. You may need to configure the target endpoint if it's not the application's default.
29 +You could use this technique to monitor an entire infrastructure of Docker containers. The same [enable and configure](/src/collectors/REFERENCE.md) procedures apply whether an application runs on the host system or inside a container. You may need to configure the target endpoint if it's not the application's default.
30
61 -Netdata can even [run in a Docker container](/packaging/docker/README.md) itself, and then collect metrics about the
62 -host system, its own container with cgroups, and any applications you want to monitor.
31 +Netdata can even [run in a Docker container](/packaging/docker/README.md) itself, and then collect metrics about the host system, its own container with cgroups, and any applications you want to monitor.
32
64 -See our [application metrics doc](/docs/collecting-metrics/application-metrics.md) for details about Netdata's application metrics
65 -collection capabilities.
33 +See our [application metrics doc](/docs/collecting-metrics/application-metrics.md) for details about Netdata's application metrics collection capabilities.
34
35 ## Collect Kubernetes metrics
36
69 -We already have a few complementary tools and collectors for monitoring the many layers of a Kubernetes cluster,
70 -_entirely for free_. These methods work together to help you troubleshoot performance or availability issues across
71 -your k8s infrastructure.
72 -
73 -- A [Helm chart](https://github.com/netdata/helmchart), which bootstraps a Netdata Agent pod on every node in your
74 - cluster, plus an additional parent pod for storing metrics and managing alert notifications.
75 -- A [service discovery plugin](https://github.com/netdata/agent-service-discovery), which discovers and creates
76 - configuration files for [compatible
77 - applications](https://github.com/netdata/helmchart#service-discovery-and-supported-services) and any endpoints
78 - covered by our [generic Prometheus
79 - collector](/src/go/collectors/go.d.plugin/modules/prometheus/README.md). With these
80 - configuration files, Netdata collects metrics from any compatible applications as they run _inside_ a pod.
81 - Service discovery happens without manual intervention as pods are created, destroyed, or moved between nodes.
82 -- A [Kubelet collector](/src/go/collectors/go.d.plugin/modules/k8s_kubelet/README.md), which runs
83 - on each node in a k8s cluster to monitor the number of pods/containers, the volume of operations on each container,
84 - and more.
85 -- A [kube-proxy collector](/src/go/collectors/go.d.plugin/modules/k8s_kubeproxy/README.md), which
86 - also runs on each node and monitors latency and the volume of HTTP requests to the proxy.
87 -- A [cgroups collector](/src/collectors/cgroups.plugin/README.md), which collects CPU, memory, and bandwidth metrics for
88 - each container running on your k8s cluster.
89 -
90 -For a holistic view of Netdata's Kubernetes monitoring capabilities, see our guide: [_Monitor a Kubernetes (k8s) cluster
91 -with Netdata_](/docs/developer-and-contributor-corner/kubernetes-k8s-netdata.md).
92 -
93 -## What's next?
94 -
95 -Netdata is capable of collecting metrics from hundreds of applications, such as web servers, databases, messaging
96 -brokers, and more. See more in the [application metrics doc](/docs/collecting-metrics/application-metrics.md).
97 -
98 -If you already have all the information you need about collecting metrics, move into Netdata's meaningful visualizations
99 -with [seeing an overview of your infrastructure](/docs/dashboards-and-charts/home-tab.md) using Netdata Cloud.
100 -
37 +Netdata provides a host of tools and collectors for monitoring the many layers of a Kubernetes cluster. These methods work together to help you troubleshoot performance or availability issues across your k8s infrastructure.
38
39 +- A [Helm chart](https://github.com/netdata/helmchart), which bootstraps a Netdata Agent pod on every node in your cluster, plus an additional parent pod for storing metrics and managing alert notifications.
40 +- A [service discovery plugin](https://github.com/netdata/agent-service-discovery), which discovers and creates configuration files for [compatible applications](https://github.com/netdata/helmchart#service-discovery-and-supported-services) and any endpoints covered by our [generic Prometheus collector](/src/go/collectors/go.d.plugin/modules/prometheus/README.md). With these configuration files, Netdata collects metrics from any compatible applications as they run _inside_ a pod. Service discovery happens without manual intervention as pods are created, destroyed, or moved between nodes.
41 +- A [Kubelet collector](/src/go/collectors/go.d.plugin/modules/k8s_kubelet/README.md), which runs on each node in a k8s cluster to monitor the number of pods/containers, the volume of operations on each container, and more.
42 +- A [kube-proxy collector](/src/go/collectors/go.d.plugin/modules/k8s_kubeproxy/README.md), which also runs on each node and monitors latency and the volume of HTTP requests to the proxy.
43 +- A [cgroups collector](/src/collectors/cgroups.plugin/README.md), which collects CPU, memory, and bandwidth metrics for each container running on your k8s cluster.
docs/collecting-metrics/system-metrics.md
+7 -56
@@ -1,62 +1,13 @@
1 -<!--
2 -title: "Collect system metrics with Netdata"
3 -sidebar_label: "System metrics"
4 -description: "Netdata collects thousands of metrics from physical and virtual systems, IoT/edge devices, and containers with zero configuration."
5 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/collecting-metrics/system-metrics.md"
6 -learn_status: "Published"
7 -learn_topic_type: "Concepts"
8 -learn_rel_path: "Concepts"
9 --->
1 +# Collect System Metrics with Netdata
2
11 -# Collect system metrics with Netdata
3 +Netdata collects thousands of metrics directly from the operating systems of physical and virtual machines, IoT/edge devices, and [containers](/docs/collecting-metrics/container-metrics.md) with zero configuration.
4
13 -Netdata collects thousands of metrics directly from the operating systems of physical and virtual systems, IoT/edge
14 -devices, and [containers](/docs/collecting-metrics/container-metrics.md) with zero configuration.
5 +To gather system metrics, Netdata uses various plugins, each of which has one or more collectors for very specific metrics exposed by the host. The system metrics Netdata users interact with most for health monitoring and performance troubleshooting are collected and visualized by `proc.plugin`, `cgroups.plugin`, and `ebpf.plugin`.
6
16 -To gather system metrics, Netdata uses roughly a dozen plugins, each of which has one or more collectors for very
17 -specific metrics exposed by the host. The system metrics Netdata users interact with most for health monitoring and
18 -performance troubleshooting are collected and visualized by `proc.plugin`, `cgroups.plugin`, and `ebpf.plugin`.
7 +[**proc.plugin**](/src/collectors/proc.plugin/README.md) gathers metrics from the `/proc` and `/sys` folders in Linux systems, along with a few other endpoints, and is responsible for the bulk of the system metrics collected and visualized by Netdata. It collects CPU, memory, disks, load, networking, mount points, and more with zero configuration. It also allows Netdata to monitor its own resource utilization.
8
20 -[**proc.plugin**](/src/collectors/proc.plugin/README.md) gathers metrics from the `/proc` and `/sys` folders in Linux
21 -systems, along with a few other endpoints, and is responsible for the bulk of the system metrics collected and
22 -visualized by Netdata. It collects CPU, memory, disks, load, networking, mount points, and more with zero configuration.
23 -It even allows Netdata to monitor its own resource utilization!
24 -
25 -[**cgroups.plugin**](/src/collectors/cgroups.plugin/README.md) collects rich metrics about containers and virtual machines
26 -using the virtual files under `/sys/fs/cgroup`. By reading cgroups, Netdata can instantly collect resource utilization
27 -metrics for systemd services, all containers (Docker, LXC, LXD, Libvirt, systemd-nspawn), and more. Learn more in the
28 -[collecting container metrics](/docs/collecting-metrics/container-metrics.md) doc.
29 -
30 -[**ebpf.plugin**](/src/collectors/ebpf.plugin/README.md): Netdata's extended Berkeley Packet Filter (eBPF) collector
31 -monitors Linux kernel-level metrics for file descriptors, virtual filesystem IO, and process management. You can use our
32 -eBPF collector to analyze how and when a process accesses files, when it makes system calls, whether it leaks memory or
33 -creating zombie processes, and more.
34 -
35 -While the above plugins and associated collectors are the most important for system metrics, there are many others. You
36 -can find all system collectors in our [supported collectors list](/src/collectors/COLLECTORS.md#system-collectors).
37 -
38 -## Collect Windows system metrics
39 -
40 -Netdata is also capable of monitoring Windows systems. The [Windows
41 -collector](/src/go/collectors/go.d.plugin/modules/windows/README.md) integrates with
42 -[windows_exporter](https://github.com/prometheus-community/windows_exporter), a small Go-based binary that you can run
43 -on Windows systems. The Windows collector then gathers metrics from an endpoint created by windows_exporter, for more
44 -details see [the requirements](/src/go/collectors/go.d.plugin/modules/windows/README.md#requirements).
45 -
46 -Next, [configure](/src/go/collectors/go.d.plugin/modules/windows/README.md#configuration) the Windows
47 -collector to point to the URL and port of your exposed endpoint. Restart Netdata with `sudo systemctl restart netdata`, or the [appropriate
48 -method](/packaging/installer/README.md#maintaining-a-netdata-agent-installation) for your system. You'll start seeing Windows system metrics, such as CPU
49 -utilization, memory, bandwidth per NIC, number of processes, and much more.
50 -
51 -For information about collecting metrics from applications _running on Windows systems_, see the [application metrics
52 -doc](/docs/collecting-metrics/application-metrics.md#collect-metrics-from-applications-running-on-windows).
53 -
54 -## What's next?
55 -
56 -Because there's some overlap between system metrics and [container metrics](/docs/collecting-metrics/container-metrics.md), you
57 -should investigate Netdata's container compatibility if you use them heavily in your infrastructure.
58 -
59 -If you don't use containers, skip ahead to collecting [application metrics](/docs/collecting-metrics/application-metrics.md) with
60 -Netdata.
9 +[**cgroups.plugin**](/src/collectors/cgroups.plugin/README.md) collects rich metrics about containers and virtual machines using the virtual files under `/sys/fs/cgroup`. By reading cgroups, Netdata can instantly collect resource utilization metrics for systemd services, all containers (Docker, LXC, LXD, Libvirt, systemd-nspawn), and more. Learn more in the [collecting container metrics](/docs/collecting-metrics/container-metrics.md) doc.
10
11 +[**ebpf.plugin**](/src/collectors/ebpf.plugin/README.md): Netdata's extended Berkeley Packet Filter (eBPF) collector monitors Linux kernel-level metrics for file descriptors, virtual filesystem IO, and process management. You can use our eBPF collector to analyze how and when a process accesses files, when it makes system calls, whether it leaks memory or creating zombie processes, and more.
12
13 +While the above plugins and associated collectors are the most important for system metrics, there are many others. You can find all of our data collection integrations [here](/src/collectors/COLLECTORS.md#system-collectors).
docs/deployment-guides/README.md
+1 -1
@@ -1,6 +1,6 @@
1 # Deployment Guides
2
3 -Netdata can be used to monitor all kinds of infrastructure, from stand-alone tiny IoT devices to complex hybrid setups combining on-premise and cloud infrastructure, mixing bare-metal servers, virtual machines and containers.
3 +Netdata can be used to monitor all kinds of infrastructure, from tiny stand-alone IoT devices to complex hybrid setups combining on-premise and cloud infrastructure, mixing bare-metal servers, virtual machines and containers.
4
5 There are 3 components to structure your Netdata ecosystem:
6
docs/deployment-guides/deployment-strategies.md
+30 -62
@@ -1,60 +1,38 @@
1 -# Deployment strategies
2 -
1 +# Deployment Examples
2
3 ## Deployment Options Overview
4
6 -This section provides a quick overview of a few common deployment options. The next sections go into configuration examples and further reading.
7 -
8 -### Stand-alone Deployment
9 -
10 -To help our users have a complete experience of Netdata when they install it for the first time, a Netdata Agent with default configuration
11 -is a complete monitoring solution out of the box, having all these features enabled and available.
12 -
13 -The Agent will act as a _stand-alone_ Agent by default, and this is great to start out with for small setups and home labs. By [connecting each Agent to Cloud](/src/claim/README.md), you can see an overview of all your nodes, with aggregated charts and centralized alerting, without setting up a Parent.
14 -
15 -![image](https://github.com/netdata/netdata/assets/116741/6a638175-aec4-4d46-85a6-520c283ab6a8)
16 -
17 -### Parent – Child Deployment
18 -
19 -An Agent connected to a Parent is called a _Child_. It will _stream_ metrics to its Parent. The Parent can then take care of storing metrics on behalf of that node (with longer retention), handle metrics queries for showing dashboards, and provide alerting.
20 -
21 -When using Cloud, it is recommended that just the Parent is connected to Cloud. Child Agents can then be configured to have short retention, in RAM instead of on Disk, and have alerting and other features disabled. Because they don't need to connect to Cloud themselves, those children can then be further secured by not allowing outbound traffic.
22 -
23 -![image](https://github.com/netdata/netdata/assets/116741/cb65698d-a6b7-43ee-a2d1-c30d0a46f084)
5 +This section provides a quick overview for a few common deployment options for Netdata.
6
25 -This setup allows for leaner Child nodes and is good for setups with more than a handful of nodes. Metrics data remains accessible if the Child node is temporarily unavailable or decommissioned, although there is no failover in case the Parent becomes unavailable.
7 +You can read about [Standalone Deployment](/docs/deployment-guides/standalone-deployment.md) and [Deployment with Centralization Points](/docs/deployment-guides/deployment-with-centralization-points.md) in the documentation inside this section.
8
9 +The sections below go into configuration examples about these deployment concepts.
10
28 -### Active–Active Parent Deployment
11 +## Deployment Configuration Details
12
30 -For high availability, Parents can be configured to stream data for their children between them, and keep the data sets in sync. Child Agents are configured with the addresses of both Parent Agents, but will only stream to one of them at a time. When that Parent becomes unavailable, it reconnects to another. When the first Parent becomes available again, that Parent will catch up by receiving the backlog from the second.
13 +### Stand-alone
14
32 -With both Parent Agents connected to Cloud, Cloud will route queries to either Parent transparently, depending on their availability. Alerts trigger on either Parent will stream to Cloud, and Cloud will deduplicate and debounce state changes to prevent spurious notifications.
15 +The stand-alone setup is configured out of the box with reasonable defaults, but please consult our [configuration documentation](/docs/netdata-agent/configuration/README.md) for details, including the overview of [common configuration changes](/docs/netdata-agent/configuration/common-configuration-changes.md).
16
34 -![image](https://github.com/netdata/netdata/assets/116741/6ae2b10c-7f7d-4503-aac4-0a9381c6f80b)
17 +### Parent – Child
18
19 +For setups involving Parent and Child Agents, they need to be configured for [streaming](docs/observability-centralization-points/metrics-centralization-points/configuration.md), through the configuration file `stream.conf`.
20
37 -## Configuration Details
21 +This will instruct the Child to stream data to the Parent and the Parent to accept streaming connections for one or more Child Agents. To secure this connection, both need a shared API key (to replace the string `API_KEY` in the examples below). Additionally, the Child can be configured with one or more addresses of Parent Agents (`PARENT_IP_ADDRESS`).
22
39 -### Stand-alone Deployment
40 -
41 -The stand-alone setup is configured out of the box with reasonable defaults, but please consult our [configuration documentation](/docs/netdata-agent/configuration/cheatsheet.md) for details, including the overview of [common configuration changes](/docs/netdata-agent/configuration/common-configuration-changes.md).
42 -
43 -### Parent – Child Deployment
44 -
45 -For setups involving Child and Parent Agents, the Agents need to be configured for [_streaming_](/src/streaming/README.md), through the configuration file `stream.conf`. This will instruct the Child to stream data to the Parent and the Parent to accept streaming connections for one or more Child Agents. To secure this connection, both need set up a shared API key (to replace the string `API_KEY` in the examples below). Additionally, the Child is configured with one or more addresses of Parent Agents (`PARENT_IP_ADDRESS`).
46 -
47 -An API key is a key created with `uuidgen` and is used for authentication and/or customization in the Parent side. I.e. a Child will stream using the API key, and a Parent is configured to accept connections from Child, but can also apply different options for children by using multiple different API keys. The easiest setup uses just one API key for all Child Agents.
23 +An API key is a key created with `uuidgen` and is used for authentication and/or customization on the Parent side. For example, a Child can stream using the API key, and a Parent can be configured to accept connections from the Child, but it can also apply different options for Children by using multiple different API keys. The easiest setup uses just one API key for all Child Agents.
24
25 #### Child config
26
51 -As mentioned above, the recommendation is to not claim the Child to Cloud directly during your setup, avoiding establishing an [ACLK](/src/aclk/README.md) connection.
27 +As mentioned above, we do not recommend to claim the Child to Cloud directly during your setup.
28
53 -To reduce the footprint of the Netdata Agent on your production system, some capabilities can be switched OFF on the Child and kept ON on the Parent. In this example, Machine Learning and Alerting are disabled in the Child, so that the Parent can take the load. We also use RAM instead of disk to store metrics with limited retention, covering temporary network issues.
29 +This is done in order to reduce the footprint of the Netdata Agent on your production system, as some capabilities can be switched OFF for the Child and kept ON for the Parent.
30 +
31 +In this example, Machine Learning and Alerting are disabled for the Child, so that the Parent can take the load. We also use RAM instead of disk to store metrics with limited retention, covering temporary network issues.
32
33 ##### netdata.conf
34
57 -On the child node, edit `netdata.conf` by using the edit-config script: `/etc/netdata/edit-config netdata.conf` set the following parameters:
35 +On the child node, edit `netdata.conf` by using the [edit-config](docs/netdata-agent/configuration/README.md#edit-netdataconf) script and set the following parameters:
36
37 ```yaml
38 [db]
@@ -85,9 +63,7 @@ On the child node, edit `netdata.conf` by using the edit-config script: `/etc/ne
63
64 ##### stream.conf
65
88 -To edit `stream.conf`, again use the edit-config script: `/etc/netdata/edit-config stream.conf`.
89 -
90 -Set the following parameters:
66 +To edit `stream.conf`, use again the [edit-config](docs/netdata-agent/configuration/README.md#edit-netdataconf) script and set the following parameters:
67
68 ```yaml
69 [stream]
@@ -101,7 +77,7 @@ Set the following parameters:
77
78 #### Parent config
79
104 -For the Parent, besides setting up streaming, the example will also provide an example configuration of multiple [tiers](/src/database/engine/README.md#tiering) of metrics [storage](/docs/netdata-agent/configuration/optimizing-metrics-database/change-metrics-storage.md), for 10 children, with about 2k metrics each.
80 +For the Parent, besides setting up streaming, this example also provides configuration for multiple [tiers of metrics storage](/docs/netdata-agent/configuration/optimizing-metrics-database/change-metrics-storage.md#calculate-the-system-resources-ram-disk-space-needed-to-store-metrics), for 10 Children, with about 2k metrics each. This allows for:
81
82 - 1s granularity at tier 0 for 1 week
83 - 1m granularity at tier 1 for 1 month
@@ -114,7 +90,7 @@ Requiring:
90
91 ##### netdata.conf
92
117 -On the Parent, edit `netdata.conf` with `/etc/netdata/edit-config netdata.conf` and set the following parameters:
93 +On the Parent, edit `netdata.conf` by using the [edit-config](docs/netdata-agent/configuration/README.md#edit-netdataconf) script and set the following parameters:
94
95 ```yaml
96 [db]
@@ -149,7 +125,7 @@ On the Parent, edit `netdata.conf` with `/etc/netdata/edit-config netdata.conf`
125
126 ##### stream.conf
127
152 -On the Parent node, edit `stream.conf` with `/etc/netdata/edit-config stream.conf`, and then set the following parameters:
128 +On the Parent node, edit `stream.conf` by using the [edit-config](docs/netdata-agent/configuration/README.md#edit-netdataconf) script and set the following parameters:
129
130 ```yaml
131 [API_KEY]
@@ -157,13 +133,13 @@ On the Parent node, edit `stream.conf` with `/etc/netdata/edit-config stream.con
133 enabled = yes
134 ```
135
160 -### Active–Active Parent Deployment
136 +### Active–Active Parents
137
162 -In order to setup active–active streaming between Parent 1 and Parent 2, Parent 1 needs to be instructed to stream data to Parent 2 and Parent 2 to stream data to Parent 1. The Child Agents need to be configured with the addresses of both Parent Agents. The Agent will only connect to one Parent at a time, falling back to the next if the previous failed. These examples use the same API key between Parent Agents as for connections from Child Agents.
138 +In order to setup active–active streaming between Parent 1 and Parent 2, Parent 1 needs to be instructed to stream data to Parent 2 and Parent 2 to stream data to Parent 1. The Child Agents need to be configured with the addresses of both Parent Agents. An Agent will only connect to one Parent at a time, falling back to the next upon failure. These examples use the same API key between Parent Agents and for connections for Child Agents.
139
164 -On both Netdata Parent and all Child Agents, edit `stream.conf` with `/etc/netdata/edit-config stream.conf`:
140 +On both Netdata Parent and all Child Agents, edit `stream.conf` by using the [edit-config](docs/netdata-agent/configuration/README.md#edit-netdataconf) script:
141
166 -##### stream.conf on Parent 1
142 +#### stream.conf on Parent 1
143
144 ```yaml
145 [stream]
@@ -178,7 +154,7 @@ On both Netdata Parent and all Child Agents, edit `stream.conf` with `/etc/netda
154 enabled = yes
155 ```
156
181 -##### stream.conf on Parent 2
157 +#### stream.conf on Parent 2
158
159 ```yaml
160 [stream]
@@ -192,7 +168,7 @@ On both Netdata Parent and all Child Agents, edit `stream.conf` with `/etc/netda
168 enabled = yes
169 ```
170
195 -##### stream.conf on Child Agents
171 +#### stream.conf on Child Agents
172
173 ```yaml
174 [stream]
@@ -208,19 +184,11 @@ On both Netdata Parent and all Child Agents, edit `stream.conf` with `/etc/netda
184
185 We strongly recommend the following configuration changes for production deployments:
186
211 -1. Understand Netdata's [security and privacy design](/docs/security-and-privacy-design/README.md) and
212 - [secure your nodes](/docs/netdata-agent/securing-netdata-agents.md)
187 +1. Understand Netdata's [security and privacy design](/docs/security-and-privacy-design/README.md) and [secure your nodes](/docs/netdata-agent/securing-netdata-agents.md)
188
189 To safeguard your infrastructure and comply with your organization's security policies.
190
216 -2. Set up [streaming and replication](/src/streaming/README.md) to:
217 -
218 - - Offload Netdata Agents running on production systems and free system resources for the production applications running on them.
219 - - Isolate production systems from the rest of the world and improve security.
220 - - Increase data retention.
221 - - Make your data highly available.
222 -
223 -3. [Optimize the Netdata Agents system utilization and performance](/docs/netdata-agent/configuration/optimize-the-netdata-agents-performance.md)
191 +2. [Optimize the Netdata Agents system utilization and performance](/docs/netdata-agent/configuration/optimize-the-netdata-agents-performance.md)
192
193 To save valuable system resources, especially when running on weak IoT devices.
194
@@ -228,11 +196,11 @@ We also suggest that you:
196
197 1. [Use Netdata Cloud to access the dashboards](/docs/netdata-cloud/monitor-your-infrastructure.md)
198
231 - For increased security, user management and access to our latest tools for advanced dashboarding and troubleshooting.
199 + For increased security, user management and access to our latest features, tools and troubleshooting solutions.
200
201 2. [Change how long Netdata stores metrics](/docs/netdata-agent/configuration/optimizing-metrics-database/change-metrics-storage.md)
202
235 - To control Netdata's memory use, when you have a lot of ephemeral metrics.
203 + To control Netdata's memory use, when you have a lot of ephemeral metrics.
204
205 3. [Use host labels](/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts.md)
206
docs/deployment-guides/deployment-with-centralization-points.md
+13 -12
@@ -14,7 +14,7 @@ When metrics and logs are centralized, the Children are never queried for metric
14 | Unified infrastructure dashboards for logs | All logs are accessible via the same dashboard at Netdata Cloud, although they are unified per Netdata Parent |
15 | Centrally configured alerts | Yes, at Netdata Parents |
16 | Centrally dispatched alert notifications | Yes, at Netdata Cloud |
17 -| Data are exclusively on-prem | Yes, Netdata Cloud queries Netdata Agents to satisfy dashboard queries. |
17 +| Data are exclusively on-prem | Yes, Netdata Cloud queries Netdata Agents to satisfy dashboard queries. |
18
19 A configuration with 2 observability centralization points, looks like this:
20
@@ -24,7 +24,7 @@ flowchart LR
24 dashboard
25 for all nodes"]]
26 NC(["<b>Netdata Cloud</b>
27 - decides which agents
27 + decides which Agents
28 need to be queried"])
29 SA1["Netdata at AWS
30 A1"]
@@ -93,16 +93,24 @@ flowchart LR
93 SB1 & SB2 & SBN ---|stream| PB
94 ```
95
96 -### Configuration steps for deploying Netdata with Observability Centralization Points
96 +## Active–Active Parent Deployment
97 +
98 +For high availability, Parents can be configured to stream data for their Children between them, and keep their data sets in sync. Children are configured with the addresses of both Parents, but will only stream to one of them at a time. When one Parent becomes unavailable, the Child reconnects to the other. When the first Parent becomes available again, that Parent will catch up by receiving the backlog from the second.
99 +
100 +With both Parent Agents connected to Netdata Cloud, it will route queries to either of them transparently, depending on their availability. Alerts trigger on either Parent will stream to Cloud, and Cloud will deduplicate and debounce state changes to prevent spurious notifications.
101 +
102 +## Configuration steps for deploying Netdata with Observability Centralization Points
103
104 For Metrics:
105
100 -- Install Netdata agents on all systems and the Netdata Parents.
106 +- Install Netdata Agents on all systems and the Netdata Parents.
107
108 - Configure `stream.conf` at the Netdata Parents to enable streaming access with an API key.
109
110 - Configure `stream.conf` at the Netdata Children to enable streaming to the configured Netdata Parents.
111
112 +Check the [related section in our documentation](/docs/observability-centralization-points/metrics-centralization-points/README.md) for more info
113 +
114 For Logs:
115
116 - Install `systemd-journal-remote` on all systems and the Netdata Parents.
@@ -111,11 +119,4 @@ For Logs:
119
120 - Configure `systemd-journal-upload` at the Netdata Children to enable transmission of their logs to the Netdata Parents.
121
114 -Optionally:
115 -
116 -- Disable ML, health checks and dashboard access at Netdata Children to save resources and avoid duplicate notifications.
117 -
118 -When using Netdata Cloud:
119 -
120 -- Optionally: disable dashboard access on all Netdata agents (including Netdata Parents).
121 -- Optionally: disable alert notifications on all Netdata agents (including Netdata Parents).
122 +Check the [related section in our documentation](/docs/observability-centralization-points/logs-centralization-points-with-systemd-journald/README.md) for more info
docs/deployment-guides/standalone-deployment.md
+22 -20
@@ -1,22 +1,22 @@
1 # Standalone Deployment
2
3 -To help our users have a complete experience of Netdata when they install it for the first time, a Netdata Agent with default configuration is a complete monitoring solution out of the box, having all its features enabled and available.
3 +To help our users have a complete experience of Netdata when they install it for the first time, the Netdata Agent with default configuration is a complete monitoring solution out of the box, with features enabled and available.
4
5 -So, each Netdata agent acts as a standalone monitoring system by default.
5 +So, each Netdata Agent acts as a standalone monitoring system by default.
6
7 -## Standalone agents, without Netdata Cloud
7 +## Standalone Agents, without Netdata Cloud
8
9 | Feature | How it works |
10 |:---------------------------------------------:|:----------------------------------------------------:|
11 -| Unified infrastructure dashboards for metrics | No, each Netdata agent provides its own dashboard |
12 -| Unified infrastructure dashboards for logs | No, each Netdata agent exposes its own logs |
11 +| Unified infrastructure dashboards for metrics | No, each Netdata Agent provides its own dashboard |
12 +| Unified infrastructure dashboards for logs | No, each Netdata Agent exposes its own logs |
13 | Centrally configured alerts | No, each Netdata has its own alerts configuration |
14 -| Centrally dispatched alert notifications | No, each Netdata agent sends notifications by itself |
14 +| Centrally dispatched alert notifications | No, each Netdata Agent sends notifications by itself |
15 | Data are exclusively on-prem | Yes |
16
17 -When using Standalone Netdata agents, each of them offers an API and a dashboard, at its own unique URL, that looks like `http://agent-ip:19999`.
17 +When using Standalone Netdata Agents, each of them offers an API and a dashboard, at its own unique URL, that looks like `http://agent-ip:19999`.
18
19 -So, each of the Netdata agents has to be accessed individually and independently of the others:
19 +So, each of the Netdata Agents has to be accessed individually and independently of the others:
20
21 ```mermaid
22 flowchart LR
@@ -37,7 +37,7 @@ flowchart LR
37 WEB -->|URL N| SN
38 ```
39
40 -The same is true for alert notifications. Each of the Netdata agents runs its own alerts and sends notifications by itself, according to its configuration:
40 +The same is true for alert notifications. Each of the Netdata Agents runs its own alerts and sends notifications by itself, according to its configuration:
41
42 ```mermaid
43 flowchart LR
@@ -61,23 +61,23 @@ flowchart LR
61 S1 & S2 & SN ==> OTHER
62 ```
63
64 -### Configuration steps for standalone Netdata agents without Netdata Cloud
64 +### Configuration steps for standalone Netdata Agents without Netdata Cloud
65
66 No special configuration needed.
67
68 -- Install Netdata agents on all your systems, then access each of them via its own unique URL, that looks like `http://agent-ip:19999/`.
68 +- Install Netdata Agents on all your systems, then access each of them via its own unique URL, that looks like `http://agent-ip:19999/`.
69
70 -## Standalone agents, with Netdata Cloud
70 +## Standalone Agents, with Netdata Cloud
71
72 | Feature | How it works |
73 |:---------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
74 | Unified infrastructure dashboards for metrics | Yes, via Netdata Cloud, all charts aggregate metrics from all servers. |
75 | Unified infrastructure dashboards for logs | All logs are accessible via the same dashboard at Netdata Cloud, although they are not unified (ie. logs from different servers are not multiplexed into a single view) |
76 -| Centrally configured alerts | No, each Netdata has its own alerts configuration |
76 +| Centrally configured alerts | No, each Netdata has its own alerts configuration |
77 | Centrally dispatched alert notifications | Yes, via Netdata Cloud |
78 | Data are exclusively on-prem | Yes, Netdata Cloud queries Netdata Agents to satisfy dashboard queries. |
79
80 -By [connecting all Netdata agents to Netdata Cloud](/src/claim/README.md), you can have a unified infrastructure view of all your nodes, with aggregated charts, without configuring [observability centralization points](/docs/observability-centralization-points/README.md).
80 +By [connecting all Netdata Agents to Netdata Cloud](/src/claim/README.md), you can have a unified infrastructure view of all your nodes, with aggregated charts, without configuring [observability centralization points](/docs/observability-centralization-points/README.md).
81
82 ```mermaid
83 flowchart LR
@@ -85,7 +85,7 @@ flowchart LR
85 dashboard
86 for all nodes"]]
87 NC(["<b>Netdata Cloud</b>
88 - decides which agents
88 + decides which Agents
89 need to be queried"])
90 S1["Standalone
91 Netdata
@@ -100,7 +100,7 @@ flowchart LR
100 NC -->|queries| S1 & S2 & SN
101 ```
102
103 -Similarly for alerts, Netdata Cloud receives all alert transitions from all agents, decides which notifications should be sent and how, applies silencing rules, maintenance windows and based on each Netdata Cloud space and user settings, dispatches notifications:
103 +Similarly for alerts, Netdata Cloud receives all alert transitions from all Agents, decides which notifications should be sent and how, applies silencing rules, maintenance windows and based on each Netdata Cloud space and user settings, dispatches notifications:
104
105 ```mermaid
106 flowchart LR
@@ -128,12 +128,14 @@ flowchart LR
128 S1 & S2 & SN -->|alert transition| NC
129 ```
130
131 -> Note that alerts are still triggered by Netdata agents. Netdata Cloud takes care of the notifications only.
131 +> **Note**
132 +>
133 +> Alerts are still triggered by Netdata Agents. Netdata Cloud only takes care of the notifications.
134
133 -### Configuration steps for standalone Netdata agents with Netdata Cloud
135 +### Configuration steps for standalone Netdata Agents with Netdata Cloud
136
135 -- Install Netdata agents using the commands given by Netdata Cloud, so that they will be automatically added to your Netdata Cloud space. Otherwise, install Netdata agents and then claim them via the command line or their dashboard.
137 +- Install Netdata Agents using the commands given by Netdata Cloud, so that they will be automatically connected to your Netdata Cloud space. Otherwise, install Netdata Agents and then claim them via the command line or their dashboard.
138
139 - Optionally: disable their direct dashboard access to secure them.
140
139 -- Optionally: disable their alert notifications to avoid receiving email notifications directly from them (email notifications are automatically enabled when a working MTA is found on the systems Netdata agents are installed).
141 +- Optionally: disable their alert notifications to avoid receiving email notifications directly from them (email notifications are automatically enabled when a working MTA is found on the systems Netdata Agents are installed).