@cryptotaxi247 / netdata-1 / commits / 4698d9a6f

docs: Update ToC - add realtime-monitoring and reorganize functions (#21012)

* docs: Update ToC - add realtime-monitoring and reorganize functions - Add docs/realtime-monitoring.md under 'Welcome to Netdata' section - Rename 'Top Monitoring (Netdata Functions)' to 'Top Consumers (Functions)' - Add docs/functions/processes.md to 'Top Consumers (Functions)' section * added fleet deployment and configuration management

Costa Tsaousis committed Sep 20, 2025 at 19:26 UTC 4698d9a6fdb83b4042c5f796a01f5296ad102a67
3 files changed +622 -1
docs/.map/map.csv
+4 -1
@@ -1,6 +1,8 @@
1 custom_edit_url,sidebar_label,learn_status,learn_rel_path,description
2 https://github.com/netdata/netdata/edit/master/docs/welcome-to-netdata.md,Welcome to Netdata,Published,Welcome to Netdata,
3 https://github.com/netdata/netdata/edit/master/docs/netdata-enterprise-evaluation-corrected.md,Enterprise Evaluation Guide,Published,Welcome to Netdata,
4 +https://github.com/netdata/netdata/edit/master/docs/realtime-monitoring.md,Real-time Monitoring,Published,Welcome to Netdata,
5 +https://github.com/netdata/netdata/edit/master/docs/fleet-configuration-management.md,Fleet Deployment and Configuration Management,Published,Welcome to Netdata,
6 https://github.com/netdata/netdata/edit/master/docs/getting-started-netdata/guide.md,Getting Started,Published,root,
7 https://github.com/netdata/netdata/edit/master/docs/Demo-Sites.md,Live Demo,Published,root,
8 ,,,,
@@ -143,7 +145,8 @@ https://github.com/netdata/netdata/edit/master/docs/observability-centralization
145 https://github.com/netdata/netdata/edit/master/src/collectors/systemd-journal.plugin/active_journal_centralization_guide_no_encryption.md,Active journal source without encryption,Published,Logs/Logs Centralization Points with systemd-journald,
146 ,,,,
147 ,,,,
146 -https://github.com/netdata/netdata/edit/master/docs/top-monitoring-netdata-functions.md,Top Monitoring (Netdata Functions),Published,root,Present the Netdata Functions what these are and why they should be used.
148 +https://github.com/netdata/netdata/edit/master/docs/top-monitoring-netdata-functions.md,Top Consumers (Functions),Published,root,Present the Netdata Functions what these are and why they should be used.
149 +https://github.com/netdata/netdata/edit/master/docs/functions/processes.md,Processes,Published,Top Consumers (Functions),
150 ,,,,
151 https://github.com/netdata/netdata/edit/master/src/health/README.md,Alerts & Notifications,Published,Alerts & Notifications,
152 https://github.com/netdata/netdata/edit/master/docs/alerts-and-notifications/creating-alerts-with-netdata-alerts-configuration-manager.md,Creating Alerts with the Alerts Configuration Manager,Published,Alerts & Notifications,
docs/fleet-configuration-management.md new
+315
@@ -0,0 +1,315 @@
1 +# Fleet Deployment and Configuration Management
2 +
3 +As infrastructures grow from a handful of servers to thousands of nodes across mixed environments (Linux, Kubernetes, Windows, macOS, FreeBSD), managing observability agents becomes one of the most painful operational tasks.
4 +
5 +**Without a coherent strategy, teams face:**
6 +- Dozens of exporters or collectors, each with its own configs and update cycles
7 +- Manual configuration files scattered across nodes
8 +- Service discovery gaps that lead to blind spots
9 +- Downtime during upgrades and redeployments
10 +- Compliance challenges when configuration states drift
11 +
12 +## How Netdata Solves It
13 +
14 +**Netdata's Fleet Management Philosophy:**
15 +- **Zero-Configuration**: We eliminate 99% of manual configuration through comprehensive auto-discovery on supported platforms
16 +- **Single Agent**: The Netdata Agent replaces dozens of exporters, dramatically reducing configuration complexity
17 +- **Automatic Everything**: Installation and updates are fully automated when possible, with zero-downtime updates and automatic rollback
18 +- **Flexible Management**: Choose Infrastructure as Code (IaC) for compliance, Dynamic Configuration for agility, or combine both
19 +- **Strong Backwards Compatibility**: Ensures upgrades don't break existing configurations and data - Netdata maintains compatibility across versions allowing seamless updates
20 +
21 +## Platform Capabilities
22 +
23 +| Capability | Linux | Kubernetes | FreeBSD | macOS | Windows |
24 +| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | -------------- | -------------- | -------------------------------------------------------------------------------------- |
25 +| **Auto-deploy** | ✅ [kickstart.sh](https://learn.netdata.cloud/docs/netdata-agent/installation/one-line-installer-for-all-linux-systems)¹ | ✅ [Helm](https://learn.netdata.cloud/docs/netdata-agent/installation/kubernetes) | ✅ kickstart.sh | ✅ kickstart.sh | ✅ [MSI](https://learn.netdata.cloud/docs/netdata-agent/installation/windows)² (silent) |
26 +| **Auto-update** | ✅ Built-in | ✅ Built-in | ✅ Built-in | ✅ Built-in | ⚠️ Manual³ |
27 +| **Auto-discover system metrics** | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |
28 +| **Auto-discover all processes** | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |
29 +| **Auto-discover containers & VMs** | ✅ Yes | ✅ Yes | ❌ No | ❌ No | ✅ Hyper-V |
30 +| **Auto-discover Docker apps** | ✅ Yes | ✅ Via k8s⁴ | ✅ If Docker | ✅ If Docker | ✅ If Docker |
31 +| **Auto-discover system services** | ✅ systemd | ✅ Yes | ⚠️ Limited | ⚠️ launchd | ✅ Windows Services |
32 +| **Auto-discover enterprise apps** | ✅ netlistensd⁵ | ✅ Via k8s | ❌ Manual | ❌ Manual | ✅ perflib⁶ |
33 +| **Infrastructure as Code** | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |
34 +| **Dynamic Configuration** | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |
35 +
36 +**Legend**: ✅ Full support | ⚠️ Partial support | ❌ Not available
37 +
38 +**Footnotes**:
39 +1. **kickstart.sh**: [Universal installer script](https://learn.netdata.cloud/docs/netdata-agent/installation/one-line-installer-for-all-linux-systems) that auto-detects the best installation method
40 +2. **MSI**: Microsoft Software Installer package for Windows deployment
41 +3. **Manual**: Auto-updates coming Q3 2025; currently requires PowerShell/SCCM/GPO automation
42 +4. **Via k8s**: Uses Kubernetes API for service discovery instead of Docker API
43 +5. **netlistensd**: Local network service discovery - scans for listening services on Linux
44 +6. **perflib**: Windows Performance Library - provides metrics for Windows applications
45 +
46 +## Configuration Management Paradigms
47 +
48 +The observability industry uses two primary approaches for configuration management:
49 +
50 +### Infrastructure as Code (IaC)
51 +
52 +IaC treats configurations as code artifacts that can be versioned, reviewed, and deployed through automated processes.
53 +
54 +**Common tools**:
55 +- **Ansible**: Agentless automation using YAML playbooks
56 +- **Terraform**: Declarative infrastructure provisioning
57 +- **Puppet/Chef**: Agent-based configuration management
58 +- **Salt**: Event-driven automation platform
59 +
60 +**Typical workflow**:
61 +1. Define configuration in code
62 +2. Store in version control
63 +3. Review through pull requests
64 +4. Deploy via CI/CD pipeline
65 +5. Validate deployment state
66 +
67 +### Dynamic Configuration Management
68 +
69 +Dynamic configuration uses a central control plane to manage configurations without requiring code deployments.
70 +
71 +**Common implementations**:
72 +- Web-based configuration interfaces
73 +- API-driven configuration updates
74 +- Real-time configuration synchronization
75 +- Central configuration databases
76 +
77 +**Typical workflow**:
78 +1. Access central management interface
79 +2. Modify configuration through UI/API
80 +3. Changes propagate to agents
81 +4. Validation occurs at edge nodes
82 +5. Status reported back to central system
83 +
84 +### What to Use
85 +
86 +| Approach | When to Use | How |
87 +| ----------------- | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
88 +| **IaC Only** | Compliance requirements, audit trails | Ansible, Terraform, Puppet, Chef |
89 +| **Dynamic Only** | Small teams, rapid iteration | Netdata Cloud UI or REST API |
90 +| **Hybrid (Best)** | Most organizations | Base config in Git, credentials/thresholds via [UI](https://learn.netdata.cloud/docs/netdata-agent/configuration/dynamic-configuration-manager) |
91 +
92 +## How Netdata Solves Deployment Challenges at Scale
93 +
94 +Netdata addresses fleet deployment through a comprehensive strategy that minimizes operational overhead while maximizing flexibility:
95 +
96 +### Universal Single-Script Installation
97 +
98 +Netdata provides a unified installation experience across all platforms (except Windows) through `kickstart.sh`, which implements an intelligent cascade:
99 +
100 +**Installation priority order on Linux**:
101 +1. **Native binary packages** - For supported distributions (RPM, DEB, SUSE packages)
102 +2. **Static builds** - Pre-compiled binaries for x86_64, armv7l, armv6l, aarch64 architectures
103 +3. **Source compilation** - Automatic build from source as final fallback
104 +
105 +**Platform-specific behavior**:
106 +- **Linux/FreeBSD/macOS**: Single command installation via `kickstart.sh`
107 +- **FreeBSD/macOS**: Direct compilation from source (no native packages or static builds)
108 +- **Windows**: Separate MSI installer (auto-updates not yet available)
109 +
110 +### Automatic Updates
111 +
112 +All Netdata installations (except Windows) auto-update to the latest version:
113 +- Updates are scheduled via systemd timers, cron, or interval scripts
114 +- Maintains the same release channel (stable/nightly) as initially installed
115 +- Zero-downtime updates with automatic rollback on failure
116 +- No manual intervention required for security patches and new features
117 +
118 +**Strong backwards compatibility**:
119 +Netdata ensures upgrades don't break existing configurations and data - Netdata maintains compatibility across versions allowing seamless updates (see [Netdata Infrastructure](https://learn.netdata.cloud/docs/welcome-to-netdata) for architectural details)
120 +
121 +## How Netdata Solves Fleet Configuration Management at Scale
122 +
123 +Netdata supports both IaC and Dynamic Configuration while minimizing the need for manual configuration through extensive auto-discovery capabilities.
124 +
125 +### Auto-Discovery and Zero Configuration
126 +
127 +Netdata's primary operational approach is to eliminate manual configuration through comprehensive auto-discovery:
128 +- **On Linux**: Achieves true zero-configuration - 99% of services are automatically detected and monitored
129 +- **On Kubernetes**: Uses Kubernetes API for comprehensive service discovery
130 +- **On Windows**: Auto-discovers enterprise applications via perflib
131 +- **On FreeBSD/macOS**: System metrics are automatic, but application monitoring requires manual configuration
132 +- Discovers new services at runtime through periodic scanning (where supported)
133 +
134 +### Infrastructure as Code (IaC) Support
135 +
136 +For organizations with established DevOps practices, Netdata fully supports configuration management through traditional IaC tools like [Ansible](https://learn.netdata.cloud/docs/netdata-agent/installation/ansible), allowing version-controlled, auditable deployments.
137 +
138 +### Dynamic Configuration via the dashboard
139 +
140 +Through [Netdata Cloud and the Dynamic Configuration Manager](https://learn.netdata.cloud/docs/netdata-agent/configuration/dynamic-configuration-manager) and authenticated Netdata Agent and Parent dashboards (the user must sign-in to the dashboard), users can manage collector configurations and alert rules across their entire fleet without touching configuration files, restarting or redeploying agents.
141 +
142 +### Configuration Priority Order
143 +
144 +When multiple configuration sources exist for the same component, Netdata applies them in the following priority order (highest to lowest):
145 +
146 +1. **[Dynamic Configuration (DynCfg)](https://learn.netdata.cloud/docs/netdata-agent/configuration/dynamic-configuration-manager)** - Runtime configurations via Netdata Cloud UI or API
147 +2. **[User Configuration](https://learn.netdata.cloud/docs/netdata-agent/configuration/configuration)** - Files in `/etc/netdata` or `/opt/netdata/etc/netdata`
148 +3. **Auto-discovered Configuration** - Settings from service discovery mechanisms
149 +4. **Stock Configuration** - Default configuration files shipped with Netdata
150 +5. **Internal Defaults** - Built-in defaults in the code
151 +
152 +## Automatic Discovery
153 +
154 +### Operating System Metrics
155 +
156 +Netdata auto-detects operating system metrics (compute, memory, networking stack, storage, etc) on all platforms.
157 +
158 +On Linux, Netdata autodetect all kernel modules and technologies which have been instrumented, including firewalls, DDoS protections systems, storage technologies and filesystems, etc. Usually all these technologies require zero configuration.
159 +
160 +Similarly for Windows, Netdata will autodetect everything exposed via Perflib.
161 +
162 +### [Process and Application Monitoring](https://learn.netdata.cloud/docs/collecting-metrics/processes-and-system-services/applications) (apps.plugin)
163 +The apps.plugin provides intelligent process tree aggregation and monitoring on all platforms (Linux, FreeBSD, macOS, Windows):
164 +
165 +**Intelligent Process Tree Aggregation**:
166 +- Automatically traverses the entire process tree to understand process relationships
167 +- Identifies process managers (spawn servers/orchestrators like systemd, docker, containerd, etc.)
168 +- Creates a finite, manageable set of metrics by intelligently grouping the entire process tree
169 +
170 +**Resource Monitoring** (per application group):
171 +- CPU utilization (user/system, context switches)
172 +- Memory usage (real/virtual, page faults)
173 +- Disk I/O (physical/logical reads/writes)
174 +- Network traffic (if eBPF is enabled on Linux)
175 +- File descriptors and handles
176 +- Process/thread counts
177 +- Accumulated uptime
178 +
179 +**Key Benefits**:
180 +- Zero configuration required - starts monitoring immediately with intelligent defaults
181 +- Captures both running and exited processes, ensuring short-lived processes are accounted for
182 +- Provides instant visibility into resource usage for any application
183 +- Particularly valuable for shell scripts that spawn numerous short-lived subprocesses
184 +- On Windows, automatically monitors all processes and Windows services
185 +
186 +This provides comprehensive application monitoring even for software without specific collectors, making it an essential first line of observability.
187 +
188 +### [Applications](https://learn.netdata.cloud/docs/collecting-metrics/collectors-configuration) (go.d.plugin)
189 +The go.d.plugin provides auto-discovery for 150+ applications through multiple mechanisms:
190 +
191 +**Service discovery mechanisms**:
192 +- **Local network service discovery (netlistensd)** - Linux-only, uses `local-listeners` binary to detect listening services
193 +- **[Docker container discovery](https://learn.netdata.cloud/docs/collecting-metrics/container-services/docker) (dockersd)** - Discovers applications running in Docker containers
194 +- **[Kubernetes service discovery](https://learn.netdata.cloud/docs/netdata-agent/installation/kubernetes) (k8ssd)** - Discovers services running in Kubernetes pods
195 +- **[SNMP device discovery](https://learn.netdata.cloud/docs/collecting-metrics/network-devices/snmp) (snmpsd)** - Discovers and profiles SNMP-enabled network devices
196 +- **Configuration file scanning** - Detects applications based on their configuration files
197 +
198 +**Platform-specific behavior**:
199 +
200 +**Linux systems (non-Kubernetes)**:
201 +Full auto-discovery is available through the `local-listeners` utility which:
202 +- Scans for TCP/UDP sockets in LISTEN state every 2 minutes
203 +- Identifies services by their listening ports and process information
204 +- Maintains a cache with 10-minute expiry for discovered services
205 +- Automatically creates collector jobs for recognized services
206 +- Docker container discovery (dockersd) is enabled
207 +
208 +**Non-Linux platforms (FreeBSD, macOS, Windows)**:
209 +- Auto-discovery via netlistensd is **NOT available**
210 +- Docker container discovery (dockersd) works if Docker is available
211 +
212 +**Discovery status management**:
213 +When services are discovered but cannot be monitored, the Dynamic Configuration (DynCfg) system tracks their status:
214 +- **failed** - Collector failed to connect or collect data
215 +- **incomplete** - Configuration requires additional parameters (e.g., credentials)
216 +
217 +Users can view discovered services in the Netdata Cloud UI and supply missing credentials or configuration parameters through the interface, allowing the collectors to retry connection without manual file editing and without restarting Netdata.
218 +
219 +Supported applications include databases (MySQL, PostgreSQL, Redis, MongoDB), web servers (NGINX, Apache, HAProxy), message queues (RabbitMQ, Kafka), SNMP, and [many more](https://learn.netdata.cloud/docs/collecting-metrics/collectors-configuration).
220 +
221 +### Microsoft Windows
222 +
223 +**Installation**: Windows requires the MSI installer instead of kickstart.sh:
224 +```powershell
225 +# Silent installation for fleet deployment
226 +msiexec /i netdata-installer.msi /qn /norestart `
227 + CLAIMING_TOKEN="YOUR_TOKEN" `
228 + CLAIMING_ROOMS="YOUR_ROOM_ID" `
229 + CLAIMING_URL="https://app.netdata.cloud"
230 +
231 +# Via Group Policy or SCCM
232 +# Deploy MSI with TRANSFORMS for site-specific configuration
233 +```
234 +
235 +**Update Management** (Manual - auto-updates coming):
236 +- Use Windows Update Services (WSUS) or System Center Configuration Manager (SCCM)
237 +- PowerShell DSC (Desired State Configuration) for version enforcement
238 +- Scheduled task to check and download new versions
239 +
240 +**[Windows-Specific Auto-Discovery](https://learn.netdata.cloud/docs/netdata-agent/installation/windows)**:
241 +Windows monitoring is handled by the native **windows.plugin** which uses Windows Performance Counters (perflib) to automatically discover and monitor:
242 +
243 +**Enterprise Applications** (auto-discovered via perflib):
244 +- **IIS** (Internet Information Services) - Web server metrics, site traffic, requests, connections
245 +- **MS SQL Server** - Database performance, transactions, locks, buffer cache
246 +- **MS Exchange** - Mail server metrics, mailbox statistics, transport queues
247 +- **Active Directory** - Domain controller metrics, LDAP operations, replication
248 +- **Active Directory Certificate Services** - Certificate enrollment, requests, revocations
249 +- **Active Directory Federation Services** - Authentication metrics, token issuance
250 +- **Hyper-V** - Virtual machine metrics, CPU/memory allocation, network statistics
251 +- **ASP.NET** - Application performance, request execution, session state
252 +- **.NET Framework** - CLR performance, garbage collection, exceptions
253 +
254 +**System Monitoring** (automatically enabled):
255 +- Windows Services status and health
256 +- Process and thread statistics
257 +- Memory management and paging
258 +- Network interfaces and protocols
259 +- Physical and logical disk performance
260 +- NUMA architecture metrics
261 +- Power supply and battery status
262 +- Thermal zones and sensors
263 +- Semaphore and synchronization objects
264 +
265 +**Process Monitoring**:
266 +- apps.plugin provides detailed per-process and per-application monitoring on Windows
267 +- Automatically groups Windows services and applications
268 +- Monitors all processes for CPU, memory, handles, and I/O usage
269 +
270 +### Kubernetes Environments
271 +When Netdata runs inside a Kubernetes cluster, it provides comprehensive multi-level discovery:
272 +
273 +**Cluster Monitoring**:
274 +- **[Cluster state](https://learn.netdata.cloud/docs/netdata-agent/installation/kubernetes)** (k8s_state): Monitors nodes, pods, containers, deployments, services
275 +- **[Kubelet metrics](https://learn.netdata.cloud/docs/netdata-agent/installation/kubernetes) (k8s_kubelet)**: Container and pod resource usage, volume statistics
276 +- **[Control plane](https://learn.netdata.cloud/docs/netdata-agent/installation/kubernetes)** (kube-proxy, kube-scheduler, kube-controller-manager): When accessible
277 +
278 +**[Service Discovery](https://learn.netdata.cloud/docs/collecting-metrics/metrics-centralization-points/clustering-and-high-availability-of-netdata-parents)** (k8ssd):
279 +- Automatically enabled, replacing traditional host-based discovery
280 +- Uses Kubernetes API to monitor all pods and services
281 +- Discovers applications by inspecting pod containers and their exposed ports
282 +- Extracts environment variables, ConfigMaps, and Secrets for configuration
283 +- Groups targets by pod, service, and namespace metadata
284 +- Creates monitoring targets for each container:
285 + - One target per exposed port if ports are defined
286 + - One target using pod IP if no ports are exposed
287 +- Tags all metrics with Kubernetes metadata (namespace, pod name, labels, annotations)
288 +
289 +**Discovery Behavior in Kubernetes**:
290 +- **k8ssd** becomes the primary discovery mechanism
291 +- **dockersd** is automatically disabled to avoid conflicts
292 +- **netlistensd** (local-listeners) still works if pod has host network access
293 +- Stock configuration files are ignored (only user-provided configs are loaded)
294 +- Application metrics from Prometheus endpoints are automatically collected
295 +
296 +Most Kubernetes deployments require no configuration beyond the initial [Helm chart installation](https://learn.netdata.cloud/docs/netdata-agent/installation/kubernetes-helm-chart-reference).
297 +
298 +## Comparison with Other Platforms
299 +
300 +Understanding how different platforms handle configuration helps in planning migrations or hybrid deployments:
301 +
302 +### Configuration Requirements by Platform
303 +
304 +| Platform | Components per Node | Configuration Method | Auto-Discovery | Updates & Restarts |
305 +| ------------- | ---------------------------- | ----------------------------- | -------------- | ------------------------------------ |
306 +| Netdata | **Single agent** | Files, Ansible, or Dynamic UI | **Extensive** | **Zero-downtime for Dynamic Config** |
307 +| Prometheus | 5-20 exporters | YAML files per exporter | Limited | Rolling restarts |
308 +| Datadog | Agent + integrations | YAML files | Moderate | Agent restart |
309 +| OpenTelemetry | Collectors + instrumentation | YAML files per collector | Limited | Full restart |
310 +
311 +## Related Documentation
312 +- [Parent-Child Configuration Reference](https://learn.netdata.cloud/docs/netdata-parents/parent-child-configuration-reference)
313 +- [Parent Configuration Best Practices](https://learn.netdata.cloud/docs/netdata-parents/parent-configuration-best-practices)
314 +- [Streaming and Routing Reference](https://learn.netdata.cloud/docs/netdata-parents/streaming-routing-reference)
315 +- [Dynamic Configuration Manager](https://learn.netdata.cloud/docs/netdata-agent/configuration/dynamic-configuration-manager)
docs/realtime-monitoring.md new
+303
@@ -0,0 +1,303 @@
1 +# Real-Time Monitoring: The Netdata Standard
2 +
3 +## Executive Summary
4 +
5 +Netdata defines what real-time monitoring truly means: **1-second data collection and 1–2 second total latency from event to insight**, at scale. While most monitoring systems operate on 10–60-second intervals, Netdata provides _true_ sub-2-second visibility without overhead. This difference is critical: most operational anomalies last under 10 seconds, which means traditional monitoring misses them completely.
6 +
7 +With Netdata, organizations gain:
8 +
9 +- **Faster MTTR:** Instant feedback accelerates troubleshooting.
10 +- **Accurate Detection:** See true patterns, not averages.
11 +- **Resilience:** Catch spikes before they cascade into outages.
12 +- **Security:** Detect attacks unfolding in mere seconds.
13 +- **Operational Efficiency:** Enable smarter autoscaling and container monitoring.
14 +
15 +In short: when every second matters, Netdata is the only monitoring solution proven to deliver real-time performance at global scale.
16 +
17 +---
18 +
19 +Netdata pioneered the 1-second standard for operational monitoring, establishing what true real-time monitoring means in practice. While other monitoring solutions typically operate on 10-second, 30-second, or even minute-level intervals, Netdata has proven that sustained per-second monitoring at scale is both achievable and essential for modern operations.
20 +
21 +This document defines real-time monitoring, explains why it matters, and details how Netdata achieves true real-time performance where others cannot.
22 +
23 +## What is Real-Time Monitoring?
24 +
25 +### Industry Definitions
26 +The Webster dictionary defines _real-time_ as _“the simultaneous recording of an event with the actual occurrence of that event.”_
27 +
28 +In engineering, the meaning extends to systems that **record and respond to events instantly**.
29 +
30 +As Richard Hackathorn wrote in _[The BI Watch: Real-Time to Real-Value](https://www.researchgate.net/publication/228498840_The_BI_watch_real-time_to_real-value)_:
31 +
32 +> _The key concept behind “real time” is that our artificial representation of the world must be in sync with the real world, so that we can respond to events in an effective manner._
33 +
34 +Wikipedia defines **[real-time business intelligence](https://en.wikipedia.org/wiki/Real-time_business_intelligence)** as a range from _milliseconds to ≤ 5 seconds_ after an event has occurred, and identifies three types of latency involved:
35 +
36 +- **Data latency** — time to collect and store data
37 +- **Analysis latency** — time to process data into information
38 +- **Action latency** — time to act on the data (e.g., visualize or alert in observability)
39 +
40 +For a system to qualify as _real-time_, the **sum of all three latencies** must remain within the real-time window.
41 +
42 +### The Netdata Standard for Real-Time
43 +Based on these definitions, Netdata establishes this practical taxonomy:
44 +
45 +Classification | Total Latency | Netdata's Position
46 +---:|:---:|:---
47 +**Real-time** | ≤ 5 seconds | ✅ Netdata: 1-2 seconds total
48 +**Near real-time** | 5-30 seconds | ❌ Most "modern" monitoring tools
49 +**Not real-time** | > 30 seconds | ❌ Traditional monitoring systems
50 +
51 +**Netdata is designed to keep the sum of all three latencies under 2 seconds**, making it one of the few monitoring systems that qualifies as truly real-time at scale, under rigorous definitions.
52 +
53 +## Why Real-Time Monitoring is Non-Negotiable
54 +
55 +### The Pillars of Operational Excellence
56 +
57 +1. **Faster Mean Time to Resolution (MTTR)**
58 + Engineers see the immediate effects of their actions. When a database is slow, they alter an index and instantly observe queries running faster and resource relief. Without real-time monitoring, this iterative troubleshooting process takes 10-30x longer.
59 +2. **Accurate Incident Detection**
60 + Real-time monitoring reveals the true behavior of systems. An application using 100% CPU for 2 seconds then 0% for 8 seconds is fundamentally different from one using steady 20% CPU. Averaged metrics hide critical patterns, confuse operations teams, and delay root cause identification.
61 +3. **Preventing Cascading Failures**
62 + Problems compound exponentially. A 3-second resource spike can trigger a 30-second cascade if not caught immediately. Real-time monitoring catches the spark before it becomes a fire.
63 +4. **Security Threat Detection**
64 + Modern attacks happen in seconds: port scans (2-3 seconds), crypto-miner activation (instant CPU spikes), memory scanning attempts (burst patterns). These are invisible to systems monitoring at 30-second intervals.
65 +
66 +### Use Cases That Demand Real-Time
67 +
68 +#### Autoscaling Decisions
69 +Cloud autoscalers with 30-second visibility suffer from:
70 +
71 +- **Over-provisioning**: Delayed spike detection triggers unnecessary scaling
72 +- **Under-provisioning**: Missed micro-bursts cause user-facing degradation
73 +- **Flapping**: Slow feedback loops create oscillating scale up/down cycles
74 +
75 +With Netdata's 1-second monitoring, autoscalers see actual load patterns and make informed decisions.
76 +
77 +#### Database Performance Tuning
78 +A problematic query that runs for 3 seconds every 10 seconds shows as:
79 +
80 +- **30% constant load** with 10-second monitoring (misleading average)
81 +- **100% spike pattern** with Netdata (actual behavior)
82 +
83 +DBAs using Netdata can immediately see the effects of index changes, query plan modifications, and connection pool adjustments.
84 +
85 +#### Container and Kubernetes Monitoring
86 +Container lifespans can be seconds. Pod scheduling, startup, and shutdown events often complete in under 5 seconds. Traditional monitoring completely misses these critical events. Netdata captures the full lifecycle.
87 +
88 +#### Typical System Administration
89 +**Scenario 1: Disk throughput**
90 +An application performs disk reads at **500 MB/s for 5 seconds**, then is idle for 10 seconds. Can the application be made faster?
91 +
92 +- **Other monitoring solutions** show 15-second averages at 167MB/s. SREs: "The application can be made faster, the disks can provide up to 500MB/s. Contact the developers".
93 +- **Netdata** shows saturation for 5 seconds, then idle. SREs: "The application is already maxing the disks. Install faster disks."
94 +
95 +**Scenario 2: Network saturation**
96 +A sensitive transactional database stalls for 10 seconds every 5 minutes.
97 +
98 +- **Other monitoring solutions** (1-minute averages) show network usage rising slightly, from **200 Mb/s → 220 Mb/s** once every 5 minutes. This looks harmless, almost noise.
99 +- **Netdata** reveals the truth: a cron job runs every 5 minutes, transferring several GiB to a backup server. For those 10 seconds, the network is fully saturated, starving the database of bandwidth.
100 +
101 +With coarse averages, every team sees evidence that _someone else_ is at fault. Netdata stops the blame game by showing the **whole signal at per-second fidelity**. The debate shifts from _“who do we blame?”_ to _“how do we fix it?”_.
102 +
103 +## The Anatomy of Netdata’s Latency
104 +
105 +Netdata is designed to keep the **sum of all three latencies 1 second **. The worst case scenario for Netdata is 2 seconds:
106 +
107 +1. **Data latency:** 1 second (collection interval)
108 +2. **Analysis latency:** microseconds (negligible, CPU-speed dependent)
109 +3. **Action latency:** 1 second (visualization and alert updates)
110 +
111 +This means:
112 +
113 +- **Best case:** ~1 millisecond from event to visualization
114 +- **Worst case:** ~1999 milliseconds if collection and visualization ticks are maximally misaligned
115 +
116 +Graphically:
117 +
118 +```
119 +Data Collection Pace
120 +
121 + the interesting event
122 + | event collected
123 + | |
124 +--|XXXXXXXXXX|----------|----------|----------| <- data collection pace
125 + |--- 1s ---|--- 1s ---|--- 1s ---|--- 1s ---|
126 +
127 +Visualization Pace (a few ms missaligned to collection - worst case)
128 +
129 + UI fetches everything collected
130 + |
131 +--------|----------|XXXXXXXXXX|----------|----- <- visualization pace
132 + |
133 + the event visualized
134 +```
135 +
136 +The shaded boxes show the slices where an event may fall. Because both collection and visualization run on 1-second ticks, the event is guaranteed to be visible within 2 seconds.
137 +
138 +### Why One Second is the Ideal Standard
139 +
140 +- **The Universal Baseline:** 1-second is the native rhythm of universal console tools (`top`, `vmstat`, `iostat`).
141 +- **The Performance Sweet Spot:** Moving to sub-second intervals (e.g., 500ms) often doubles overhead for diminishing returns. One second is highly efficient and universally safe.
142 +- **Sufficient Resolution:** The vast majority of operational anomalies last multiple seconds; 1-second granularity captures them without loss of fidelity.
143 +- **Negligible Overhead:** Modern systems handle per-second sampling with ease; collecting a few thousand metrics per second consumes a trivial fraction of a single CPU core's billions of cycles.
144 +
145 +### Intentional Deviation from Per-Second Collection
146 +Netdata intentionally uses a longer collection interval _only_ in two specific scenarios:
147 +
148 +1. The underlying metric changes slowly (e.g., a temperature sensor).
149 +2. Per-second polling would place undue stress on the monitored application (e.g., a delicate legacy system).
150 +
151 +In these cases, Netdata chooses a responsible interval that balances fidelity with non-intrusiveness.
152 +
153 +### Beyond Latency: The Meaning of Gaps
154 +Netdata operates with the precision of a heartbeat. Unlike other solutions that treat missed collections as a normal network condition, Netdata is engineered as a watchdog: **metrics must be collected at their configured frequency**.
155 +
156 +A gap in the Netdata dashboard is not a visualization trick or a "network blip." It is a **persisted gap in the database** written to disk. It is a definitive signal that the system itself was under such severe stress that it could not service Netdata's lightweight collection process. If Netdata is missing samples, your application is almost certainly missing service-level objectives.
157 +
158 +This philosophy makes Netdata not just a monitoring tool, but a canary for system health itself.
159 +
160 +## Netdata's Real-Time Performance at Scale
161 +
162 +Netdata is a distributed-by-design platform. It scales horizontally by adding more Agents and streaming Parents. This architecture is key to its real-time capability: **adding more nodes does not increase the latency or impair the performance of existing ones.** Each node operates independently at its own 1-second rhythm, collaborating in real-time. A fleet of 10 nodes and a fleet of 10,000 nodes exhibit the same per-node, sub-2-second latency.
163 +
164 +## How Netdata Compares to Other Monitoring Solutions
165 +While most tools can be _configured_ for faster polling, their core architecture is not optimized for sustained, pervasive, per-second collection without excessive overhead or cost. This table reflects their typical, real-world deployment:
166 +
167 +| Monitoring Solution | Collection Interval | Real Latency | Why It's Not Real-Time |
168 +|---:|:---:|:---:|:---|
169 +| **Netdata** | 1 second | 1-2 seconds | ✅ **Defines the true real-time standard** |
170 +| **Prometheus + Grafana** | 10-30 seconds (typical) | 15-40 seconds | Pull-based scraping limits frequency |
171 +| **Datadog** | 15-60 seconds | 20-90 seconds | Agent batching + cloud processing |
172 +| **New Relic** | 60 seconds (default) | 60-120 seconds | Minute-level aggregation model |
173 +| **Grafana Cloud** | 10-30 seconds | 20-60 seconds | Prometheus-based limitations |
174 +| **CloudWatch** | 60-300 seconds | 60-360 seconds | AWS API polling constraints |
175 +| **Zabbix** | 30-60 seconds | 30-90 seconds | Server polling architecture |
176 +| **Nagios** | 60-300 seconds | 60-600 seconds | Check-based paradigm |
177 +| **Elastic Stack** | 10-30 seconds | 30-300 seconds | Ingest → Index → Query pipeline |
178 +| **Dynatrace** | 10-60 seconds | 15-90 seconds | Cloud analysis latency |
179 +| **AppDynamics** | 60 seconds | 60-180 seconds | Minute-level business focus |
180 +| **Splunk** | 30-300 seconds | 60-600 seconds | Log indexing architecture |
181 +
182 +### The Prometheus + Grafana Reality
183 +- Designed around 10-30 second scrape intervals; 1-second scraping is anti-pattern that overloads targets.
184 +- Pull model adds network latency to every collection
185 +- Grafana adds additional query latency on top
186 +- **Verdict**: Excellent for historical trending and long-term data analysis, but architecturally blind to micro-events. Not real-time.
187 +
188 +**Could Prometheus Be Real-Time?**
189 +Technically, yes, but practically, no. To match Netdata's data density:
190 +
191 +1. **Scalability Collapse:** Would require an order of magnitude more Prometheus servers, dramatically increasing cost and complexity.
192 +2. **Target Assault:** Frequent scraping would overwhelm the very applications it's meant to monitor.
193 +3. **Storage Inefficiency:** The Prometheus TSDB isn't optimized for this volume of per-second data.
194 +
195 +Our [stress test](https://www.netdata.cloud/blog/netdata-vs-prometheus-2025/) against a Prometheus stack at 4.6M metrics/s proved this: Netdata used **-37% CPU, -88% RAM, -13% bandwidth, and -97% disk I/O** while providing **40x longer retention** and **22x faster queries**.
196 +
197 +### The Datadog Reality
198 +- The agent defaults to 15-second intervals for most metrics.
199 +- It batches data for seconds before sending to the cloud.
200 +- Cloud processing and indexing add another 5-30 seconds of latency.
201 +- **Verdict**: A powerful analytics platform, but its architecture imposes fundamental latency constraints. It is not real-time.
202 +
203 +**Q: Datadog claims "real-time" features. How is Netdata different?**
204 +A: Datadog's "real-time" typically refers to live-tail for logs or near-live infrastructure views. Their metric pipeline latency is 20-90 seconds. Netdata provides true 1-2 second latency for all metrics, everywhere.
205 +
206 +**Q: Couldn't Datadog just collect faster?**
207 +A: No. Their business and architectural model prevents it:
208 +
209 +- **Cost Prohibitive:** 1-second collection increases metric volume 15-60x, making their pricing model untenable for customers.
210 +- **Cloud Bottlenecks:** The journey from agent to cloud to UI introduces immutable latency.
211 +- **Bandwidth:** The data transfer costs would be enormous.
212 +
213 +Netdata's edge-native architecture eliminates these bottlenecks by collecting, storing, and serving metrics locally.
214 +
215 +### The New Relic Reality
216 +- Focused on application performance (APM) with 1-minute default reporting.
217 +- Relies on sampling for high-volume events, sacrificing fidelity.
218 +- **Verdict**: Powerful for code-level application insights, but its minute-level granularity misses crucial system-level patterns. Not real-time.
219 +
220 +**Q: New Relic has 1-minute metrics. Isn't that enough?**
221 +A: Absolutely not. They hide the most critical performance patterns. A 5-second query running every 20 seconds appears as a benign 25% load average in New Relic but is immediately obvious as a damaging spike pattern in Netdata.
222 +
223 +## The Netdata Real-Time Monitoring Manifesto
224 +
225 +We believe real-time monitoring is a right, not a premium feature. Our principles are:
226 +
227 +1. **Every Second Matters:** Problems lasting 3 seconds are primary incidents, not statistical noise.
228 +2. **Gaps Are Failures:** Missing data is a symptom of system distress, not an acceptable network condition.
229 +3. **Fidelity Over Approximation:** No sampling, no averaging, no estimation - only ground truth.
230 +4. **Edge-Native Collection:** Intelligence and storage belong at the source of the data.
231 +5. **Push-Based Streaming:** Data flows outward as it happens; systems shouldn't wait to be polled.
232 +6. **Truly Horizontal Scaling:** Adding monitoring capacity must not degrade existing performance.
233 +7. **Production-Safe Efficiency:** Real-time cannot come at the cost of operational stability.
234 +
235 +## Netdata by the Numbers: The Proof is in the Performance
236 +
237 +### Per-Node Performance
238 +
239 +- **Granularity:** 1-second for all metrics, without exception.
240 +- **Volume:** 3,000-20,000+ metrics collected per second per node.
241 +- **CPU Overhead:** <5% of a single core, typical for 3,000 metrics/s.
242 +- **RAM Footprint:** <200 MB, typical for 3,000 metrics/s.
243 +- **Fault Tolerance:** Zero data loss during network issues (local buffering + replay).
244 +- **Storage Efficiency:** ~0.6 bytes per sample on disk, enabling years of retention for gigabytes, not terabytes.
245 +
246 +### Global Scale
247 +
248 +- **Billions of Metrics:** Processes over 4.5 billion metrics per second across all installations.
249 +- **100% Sampling Rate:** No statistical sampling—every data point is captured.
250 +- **Unlimited Metrics:** No artificial limits or pricing tiers based on volume.
251 +- **Proven at Scale:** Monitors infrastructures with 100,000+ nodes seamlessly.
252 +
253 +### Enterprise-Grade Reliability
254 +
255 +- **No Single Point of Failure:** Fully distributed, peer-to-peer streaming architecture.
256 +- **Automatic Failover:** Resilient streaming connections with self-healing.
257 +- **Self-Managing:** Requires minimal administration.
258 +- **Zero-Downtime Updates:** Supports rolling updates without interrupting monitoring.
259 +
260 +It is common for large Netdata deployments to process millions of metrics per second across a highly available, distributed fleet, all while remaining virtually invisible from an administrative overhead perspective. This operational elegance is Netdata's ultimate success.
261 +
262 +## Real-Time Monitoring FAQ
263 +
264 +**Q: Is Netdata truly real-time?**
265 +A: Yes. Netdata provides 1-second granularity monitoring with a total latency of 1-2 seconds from event to visualization, making it the fastest and only true real-time monitoring solution proven at scale. This is 10-60x faster than the typical "near real-time" solutions.
266 +
267 +**Q: Why is 1-second resolution critical compared to 10 or 30-second?**
268 +A: Most operational anomalies have a duration of 2-10 seconds. With 30-second monitoring, you are blind to over 90% of incidents. With 10-second monitoring, you still miss roughly 50%. Netdata's 1-second monitoring captures the full spectrum of system behavior.
269 +
270 +**Q: Doesn't per-second monitoring create unsustainable overhead?**
271 +A: No. This is a common misconception. Netdata is engineered for extreme efficiency, typically using less than 5% of a single CPU core. According to the [University of Amsterdam study](https://www.ivanomalavolta.com/files/papers/ICSOC_2023.pdf), Netdata is the most energy-efficient tool for monitoring Docker-based systems. The study also shows Netdata excels in CPU usage, RAM usage, and execution time compared to other monitoring solutions.
272 +
273 +**Q: How does Netdata handle network outages?**
274 +A: Netdata Agents buffer metrics locally on disk during network partitions and automatically replay the buffered data once the connection is restored. This ensures zero data loss. Gaps only appear if the local system is too stressed to even collect data, which is itself a critical alert. Also, each Netdata Agent and Parent provide their own dashboards allowing continue troubleshooting at extreme conditions.
275 +
276 +**Q: Can Netdata handle cloud and container environments?**
277 +A: Yes, natively. Netdata provides automatic discovery and per-second monitoring for Kubernetes, Docker, and all major cloud platforms. It collects cgroups metrics directly from the kernel. The short lifespans of containers are perfectly aligned with Netdata's real-time model.
278 +
279 +**Q: What is the core architectural difference between Netdata and Prometheus/Datadog?**
280 +A: Netdata is a distributed, edge-native system designed for real-time data. Prometheus is a pull-based centralized scrapers, and Datadog is a cloud-based SaaS platform. These fundamental models impose inherent latency that Netdata's architecture avoids entirely.
281 +
282 +**Q: Is real-time necessary for every single metric?**
283 +A: Netdata uses intelligence, not dogma. It automatically adjusts collection frequency for slow-changing metrics (like static configuration details) or for applications that are sensitive to frequent polling, while maintaining 1-second collection for all dynamic system and application metrics.
284 +
285 +**Q: Does querying more time-series slow down the dashboard?**
286 +A: Rendering thousands of lines on a chart is a browser limitation, not a Netdata limitation. For these high-cardinality views, Netdata provides instant aggregated views with the ability to drill down to specific metrics in real-time.
287 +
288 +**Q: What are the trade-offs for being real-time?**
289 +A: The trade-off is architectural complexity. Distributing intelligence to the edge, synchronizing data in real-time without central bottlenecks, and managing a fleet implicitly rather than explicitly is a significantly harder engineering problem. We solved this so you don't have to choose between real-time visibility and operational cost.
290 +
291 +## Summary
292 +
293 +Real-time monitoring is not a luxury - it is a fundamental requirement for operating modern, dynamic, and complex infrastructure. The difference between 1-second and 30-second visibility is the difference between preventing outages and merely documenting them.
294 +
295 +Netdata doesn't just claim to be real-time; it defines the category through engineering excellence:
296 +
297 +- **Bounded Latency:** Guaranteed under 2 seconds from event to insight.
298 +- **Universal 1-Second Granularity:** For all metrics, across your entire stack.
299 +- **100% Fidelity:** No averages, no samples, no approximations—only truth.
300 +- **Distributed Scale:** Provides real-time performance at any scale, from one node to one million.
301 +- **Production-Safe:** Engineered for efficiency that ensures it never becomes the problem it is designed to solve.
302 +
303 +When you need to see what is actually happening in your infrastructure right now - not a smoothed-over report of what happened a minute ago - you need Netdata. This is real-time monitoring. This is The Netdata Standard.