| 1 | # Scalability: Monitoring at Any Scale |
| 2 | |
| 3 | ## TL;DR |
| 4 | |
| 5 | Netdata scales from a single node to 100,000+ nodes without architectural changes, maintaining 1-second granularity and sub-2-second latency at any scale. The distributed edge-native architecture ensures that adding more nodes doesn't degrade performance - each node operates independently while collaborating seamlessly. |
| 6 | |
| 7 | ## The Problem with Centralization |
| 8 | |
| 9 | Traditional observability assumes one thing: push all data to a central database, then query it for dashboards and alerts. |
| 10 | |
| 11 | This works. Until it doesn't. |
| 12 | |
| 13 | When scale breaks the model, teams face two options - and both are wrong: |
| 14 | |
| 15 | ### Option 1: Reduce the Workload |
| 16 | |
| 17 | Lower granularity. Drop cardinality. Filter. Sample. |
| 18 | |
| 19 | **This is a trap and a paradox.** If you knew which data you'd need during a crisis, you could predict the crisis and prevent it. By definition, an unpredictable event is the one that will be invisible in your downsampled dataset. You're betting your incident response on being able to predict the unpredictable. |
| 20 | |
| 21 | ### Option 2: Scale the Database |
| 22 | |
| 23 | Build giant, expensive clusters. Add more cores. More RAM. More everything. |
| 24 | |
| 25 | **This is a money pit.** In many organizations today, observability costs more than the services being monitored. We routinely encounter companies where 40-50% of infrastructure budget goes to monitoring pipelines, plus teams of specialists to keep them alive. |
| 26 | |
| 27 | ## The Netdata Way: Process and Store at the Edge |
| 28 | |
| 29 | **Instead of centralizing data, distribute the code.** This is the heart of Netdata's philosophy and design. |
| 30 | |
| 31 | Every Netdata Agent is a full observability engine: |
| 32 | |
| 33 | - Collects metrics at the edge |
| 34 | - Stores data locally in multi-tier storage |
| 35 | - Runs ML-based anomaly detection in real-time |
| 36 | - Runs health checks and triggers alerts |
| 37 | - Serves dashboards and APIs independently |
| 38 | |
| 39 | When you need high availability, persistent storage for ephemeral nodes, reduced load on production systems, or on-premises dashboards, Parents aggregate streams without becoming bottlenecks - because the heavy lifting already happened at the edge. |
| 40 | |
| 41 | This distributed architecture delivers results that speak for themselves: |
| 42 | |
| 43 | - **No loss of fidelity** - every metric, every second, always visible |
| 44 | - **No blind spots** - no sampling, no cherry-picking |
| 45 | - **No scaling tax** - adding nodes adds observability, not exponential cost curves |
| 46 | |
| 47 | Once you pass ~500 nodes, you’re naturally in the multi-million metrics/s range. What looks “heroic” elsewhere is simply normal operating conditions with Netdata. |
| 48 | |
| 49 | ## Proof: The Numbers Don't Lie |
| 50 | |
| 51 | ### Independent Validation: University of Amsterdam Study (2023) |
| 52 | |
| 53 | **Study:** "An Empirical Evaluation of the Energy and Performance Overhead of Monitoring Tools on Docker-Based Systems"<br/> |
| 54 | **Conference:** ICSOC 2023 (International Conference on Service-Oriented Computing)<br/> |
| 55 | **DOI:** 10.1007/978-3-031-48421-6_13 |
| 56 | |
| 57 | **Finding:** **Netdata is the most energy-efficient monitoring solution**, with the lowest CPU and memory overhead - even while collecting data every second and running anomaly detection at the edge. |
| 58 | |
| 59 | ### Head-to-Head: Netdata vs Prometheus (2025) |
| 60 | |
| 61 | We tested a single installation Netdata Parent and Prometheus, at **4.6 million metrics per second** - the scale you hit with just 1,000 nodes. This is how the systems compare for ingestion: |
| 62 | |
| 63 | | Metric | Netdata | Prometheus | Impact | |
| 64 | |---------------------------------|-------------------|------------------|----------------------------------------------| |
| 65 | | **CPU Usage** | ~9.4 cores | ~14.8 cores | 36% less CPU | |
| 66 | | **Memory Usage** | ~47 GiB | ~383 GiB | 88% less RAM | |
| 67 | | **Disk I/O** | ~4.7 MiB/s writes | ~147 MiB/s total | 97% less I/O | |
| 68 | | **Per-second retention (1TiB)** | ~1.25 days | ~2 hours | 15x longer - 40x retention in lower tiers | |
| 69 | | **Sample completeness** | ~100% | ~93.7% | Zero data loss | |
| 70 | | **Query latency (2hr window)** | ~0.11s | ~1.8s | 16x faster - 22x faster in long term queries | |
| 71 | |
| 72 | **Critical insight:** This isn't exotic scale. **Every Netdata deployment with >500 nodes runs at millions of metrics per second.** Our users don't even notice - because the architecture absorbs it. |
| 73 | |
| 74 | ## Architecture: Built for Planet Scale |
| 75 | |
| 76 | ### Core Components |
| 77 | |
| 78 | | Component | Role | Resources (Standalone) | Resources (Offloaded) | Scale Factor | |
| 79 | |------------|-----------------------------------|-----------------------------------------|----------------------------------------|--------------------------| |
| 80 | | **Agent** | Edge collector | <5% CPU, <200 MiB RAM, disk I/O | <2% CPU, <150 MiB RAM, zero disk | 3,000-20,000 metrics/sec | |
| 81 | | **Parent** | Workload distributor & aggregator | 10 cores, 40 GiB RAM per 1M metrics/sec | Same + ML training if enabled | Linear scaling | |
| 82 | | **Cloud** | Control plane & federation | Minimal | Minimal | Unlimited Parents | |
| 83 | |
| 84 | :::note |
| 85 | |
| 86 | Agents can offload ML, alerting, dashboards, and retention to Parents - typically cutting agent CPU ≈50%, RAM ≈25%, and eliminating disk I/O entirely. |
| 87 | |
| 88 | ::: |
| 89 | |
| 90 | ### The Edge Advantage |
| 91 | |
| 92 | Each Agent is autonomous: |
| 93 | |
| 94 | - **Collects** 3,000-20,000 metrics per second per node |
| 95 | - **Stores** data in tiered storage (raw + aggregated) |
| 96 | - **Detects** anomalies using unsupervised ML |
| 97 | - **Triggers** alerts in real-time |
| 98 | - **Serves** local dashboards and APIs |
| 99 | - **Streams** to Parents for aggregation |
| 100 | |
| 101 | This means: |
| 102 | |
| 103 | - No data loss if Parent is down (Agents buffer locally) |
| 104 | - No performance degradation as you scale (work stays distributed) |
| 105 | - No architectural changes from 1 to 100,000 nodes |
| 106 | |
| 107 | ## The Parent Advantage: Intelligent Workload Distribution |
| 108 | |
| 109 | ### Why Parents Should Be Your Default |
| 110 | |
| 111 | Parents aren't just centralization points - they're intelligent workload distributors that can reduce the resource footprint on production systems. |
| 112 | |
| 113 | With Parents, Agents can offload: |
| 114 | |
| 115 | - **ML Training** - Parents train models, Agents just collect (50% CPU reduction) |
| 116 | - **Health Checks** - Parents run all alerts, Agents focus on collection |
| 117 | - **Persistent Storage** - Agents run in RAM-only mode with zero disk I/O |
| 118 | - **Dashboard Serving** - Parents handle all queries and visualizations |
| 119 | |
| 120 | A fully offloaded Agent uses <2% CPU, <150 MiB RAM, and zero disk I/O - a fraction of a standalone Agent. |
| 121 | |
| 122 | ### ML Intelligence: Train Where It Makes Sense |
| 123 | |
| 124 | Netdata's ML models flow with the metrics stream, giving you complete flexibility: |
| 125 | |
| 126 | **Option 1: ML at the Edge (default)** |
| 127 | |
| 128 | - Agents train their own models locally |
| 129 | - Models stream to Parents along with metrics |
| 130 | - Parents receive pre-computed ML results |
| 131 | - Best for: Systems with available CPU, need for immediate local anomaly detection |
| 132 | |
| 133 | **Option 2: ML at Parents** |
| 134 | |
| 135 | - Agents disable ML training (50% CPU savings) |
| 136 | - First Parent trains models for all Agents |
| 137 | - Models shared with other Parents in cluster |
| 138 | - Best for: Resource-constrained production systems, centralized ML management |
| 139 | |
| 140 | The architecture adapts to your needs - train where you have resources, use everywhere. |
| 141 | |
| 142 | ### When You Need Parents |
| 143 | |
| 144 | **We recommend Parents by default:** |
| 145 | |
| 146 | - Future-proof your architecture (same setup works at 10 or 100,000 nodes) |
| 147 | - Reduce production system load even at small scale |
| 148 | - Provide unified dashboards and centralized alerting |
| 149 | - Enable high availability and disaster recovery |
| 150 | - Cost less than the resources they save on production systems |
| 151 | |
| 152 | **Parents are essential when you have:** |
| 153 | |
| 154 | - **Ephemeral systems** - Kubernetes pods, auto-scaling VMs that disappear |
| 155 | - **Resource constraints** - Systems where every CPU cycle matters |
| 156 | - **On-premises requirements** - Multi-node view without Cloud connectivity. For a fully self-hosted Cloud experience, consider [Netdata Cloud On-Prem](https://github.com/netdata/netdata-cloud-onprem/blob/master/docs/learn.netdata.cloud/README.md) |
| 157 | - **Network restrictions** - Agents can't reach Cloud due to firewalls/policies |
| 158 | |
| 159 | ### Parent Sizing Guidelines |
| 160 | |
| 161 | | Nodes per Parent | Metrics/sec | Resources | |
| 162 | |:----------------:|:-----------:|:---------------------| |
| 163 | | ~100 nodes | ~0.5M/sec | 5 cores, 20 GiB RAM | |
| 164 | | ~250 nodes | ~1M/sec | 10 cores, 40 GiB RAM | |
| 165 | | ~500 nodes | ~2M/sec | 20 cores, 80 GiB RAM | |
| 166 | |
| 167 | :::tip |
| 168 | |
| 169 | Scale horizontally with more Parents, not vertically with bigger Parents. Beyond 500 nodes per Parent, resource usage grows non-linearly. |
| 170 | |
| 171 | ::: |
| 172 | |
| 173 | ### Parent Placement Strategy |
| 174 | |
| 175 | - **Keep Parents close** to their Agents (same datacenter, region, or cloud zone) |
| 176 | - **Minimize network hops** to reduce latency and bandwidth costs |
| 177 | - **Deploy per region** in multi-region architectures |
| 178 | - **Use multiple Parents** rather than one giant Parent |
| 179 | |
| 180 | ### High Availability & Intelligent Clustering |
| 181 | |
| 182 | Parents work together intelligently to eliminate duplicate work: |
| 183 | |
| 184 | - **Active-active Parents** with automatic work distribution |
| 185 | - **ML model sharing** - First Parent trains, others receive models |
| 186 | - **Automatic failover** - Agents reconnect to available Parents |
| 187 | - **Local buffering** - Agents retain 1+ hour of data during Parent downtime |
| 188 | - **Streaming replication** between Parents for complete redundancy |
| 189 | - **Federated queries** across all Parents via Netdata Cloud |
| 190 | |
| 191 | **Key insight:** Clustering without double-spend: In an active-active cluster, the first Parent that sees a child trains the model; peers reuse it. You get HA without multiplying heavy work. |
| 192 | |
| 193 | ### Alerts: Automation vs Monitoring |
| 194 | |
| 195 | Netdata separates automation from monitoring, letting you optimize both: |
| 196 | |
| 197 | **Agents: Local Automation** |
| 198 | |
| 199 | - Keep only alerts that trigger local scripts |
| 200 | - Example: "If CPU > 90%, scale this service" |
| 201 | - Immediate response, no network dependency |
| 202 | - Minimal overhead when selective |
| 203 | |
| 204 | **Parents: Human Monitoring** |
| 205 | |
| 206 | - Run comprehensive health checks for all Agents |
| 207 | - Send notifications to teams via Cloud or integrations |
| 208 | - Correlate issues across multiple systems |
| 209 | - Rich context for troubleshooting |
| 210 | |
| 211 | This dual approach means production systems only run automation-critical alerts while Parents handle the hundreds of monitoring alerts that humans need to see. |
| 212 | |
| 213 | ## Storage: Efficient Multi-Tier Architecture |
| 214 | |
| 215 | ### Three-Tier Storage System |
| 216 | |
| 217 | | Tier | Resolution | Compression | Retention | Use Case | |
| 218 | |------------|------------|----------------------------|-----------------|-------------------| |
| 219 | | **Tier 0** | Per-second | Minimal (0.6 bytes/sample) | Days to Weeks | Troubleshooting | |
| 220 | | **Tier 1** | Per-minute | High | Weeks to Months | Trending | |
| 221 | | **Tier 2** | Per-hour | Maximum | Months to Years | Capacity planning | |
| 222 | |
| 223 | All tiers update in parallel - no post-processing or compaction jobs needed. |
| 224 | |
| 225 | ### Storage Efficiency |
| 226 | |
| 227 | - **0.6 bytes per sample** - industry's most efficient |
| 228 | - **Gorilla + ZSTD compression** for optimal size/speed |
| 229 | - **WORM design** - append-only, no expensive compaction |
| 230 | |
| 231 | ## Why This Architecture Wins |
| 232 | |
| 233 | ### For Operations Teams |
| 234 | |
| 235 | - **No blind spots** during incidents - all data available |
| 236 | - **No architectural rewrites** as you scale |
| 237 | - **No sampling lottery** - the metric you need is always there |
| 238 | - **No specialized skills** required - it just works |
| 239 | |
| 240 | ### For Finance |
| 241 | |
| 242 | - **Predictable costs** - linear scaling, no surprises |
| 243 | - **Lower TCO** - fewer resources for same visibility |
| 244 | - **Energy efficient** - independently validated lowest overhead |
| 245 | - **Reduced team size** - less complexity to manage |
| 246 | |
| 247 | ### For Developers |
| 248 | |
| 249 | - **Per-second granularity** - see what actually happened |
| 250 | - **Real-time anomaly detection** - catch issues immediately |
| 251 | - **Local dashboards** - debug without central bottlenecks |
| 252 | - **Full cardinality** - every dimension tracked |
| 253 | |
| 254 | ## The Bottom Line |
| 255 | |
| 256 | Through intelligent workload distribution between Parents and Agents: |
| 257 | |
| 258 | - **ML trains where you have resources** (edge or Parents, your choice) |
| 259 | - **Alerts run where they matter** (automation locally, monitoring centrally) |
| 260 | - **Storage happens where it's cheap** (Parents, not production) |
| 261 | - **Millions of metrics per second is normal** (not heroic) |
| 262 | - **HA doesn't multiply overhead** (intelligent clustering) |
| 263 | |
| 264 | This isn't just optimization. It's a fundamentally different architecture that recognizes observability shouldn't compete with your applications for resources. |
| 265 | |
| 266 | **Welcome to observability that makes your infrastructure better, not heavier.** |
| 267 | |
| 268 | ## FAQ |
| 269 | |
| 270 | <details> |
| 271 | <summary><strong>How many nodes can a single Netdata Parent handle?</strong></summary><br/> |
| 272 | |
| 273 | We recommend running Parents with up to 500 Agents (1.5M metrics/s). We have customers running larger Parents, but resources increase and performance decreases non-linearly. |
| 274 | |
| 275 | </details> |
| 276 | |
| 277 | <details> |
| 278 | <summary><strong>What happens if a Parent goes down?</strong></summary><br/> |
| 279 | |
| 280 | If the Parent was clustered, agents will connect to the other Parent and replicate to it any metrics collected during the transition. If there is no other Parent to connect to, Agents keep collecting and storing data locally, which will be replicated to the Parent when it becomes available. Note that the replication of past metrics uses only tier-0 (high-res data), so Agents must have enough retention in tier-0 to avoid gaps in the charts. |
| 281 | |
| 282 | </details> |
| 283 | |
| 284 | <details> |
| 285 | <summary><strong>Do I always need Parents?</strong></summary><br/> |
| 286 | |
| 287 | No. Agents alone may be enough. Parents are usually required when you have ephemeral nodes. |
| 288 | |
| 289 | </details> |
| 290 | |
| 291 | <details> |
| 292 | <summary><strong>How much overhead does Netdata introduce on my systems?</strong></summary><br/> |
| 293 | |
| 294 | Less than 5% CPU and ~200 MiB RAM per agent in standalone mode. Offloaded agents (streaming to a Parent) drop to less than 2% CPU and ~150 MiB RAM with zero disk I/O. Netdata is designed to be "polite citizen" to production workloads, so it spreads its workload across time and avoids all kinds of sudden and intense spikes. |
| 295 | |
| 296 | </details> |
| 297 | |
| 298 | <details> |
| 299 | <summary><strong>How efficient is Netdata's storage?</strong></summary><br/> |
| 300 | |
| 301 | Tier 0 (per-second) is ~0.6 bytes/sample - the industry's most efficient. Tiers 1 & 2 keep per-minute and per-hour aggregates, letting you retain months or years of history cheaply. |
| 302 | |
| 303 | </details> |
| 304 | |
| 305 | <details> |
| 306 | <summary><strong>How do I deploy Parents in multi-region or multi-cloud setups?</strong></summary><br/> |
| 307 | |
| 308 | Place Parents close to the agents they serve (same DC/region/AZ). Deploy multiple Parents per region for HA. Use Netdata Cloud to unify dashboards and queries across Parents. |
| 309 | |
| 310 | </details> |
| 311 | |
| 312 | <details> |
| 313 | <summary><strong>What's the difference between monitoring and automation alerts?</strong></summary><br/> |
| 314 | |
| 315 | Since Netdata evaluates alerts at the edge, it allows you to specify scripts to be executed when an alert triggers. This enables automation, e.g. "restart service if API endpoint is not responding". |
| 316 | |
| 317 | </details> |
| 318 | |
| 319 | <details> |
| 320 | <summary><strong>Is Netdata really energy-efficient?</strong></summary><br/> |
| 321 | |
| 322 | Yes. A peer-reviewed 2023 study (ICSOC, University of Amsterdam) found Netdata to be the most energy-efficient tool among the ones tested, with the lowest CPU and RAM overhead even at 1-second collection. |
| 323 | |
| 324 | </details> |
| 325 | |
| 326 | <details> |
| 327 | <summary><strong>Is 100,000+ nodes single installation real?</strong></summary><br/> |
| 328 | |
| 329 | Yes. Even Netdata Cloud SaaS itself (our commercial service) is such a single installation that serves way more than 100k reachable nodes. |
| 330 | |
| 331 | </details> |
| 332 | |
| 333 | <details> |
| 334 | <summary><strong>Do you promote per-second collection and unlimited metrics because your revenue depends on volume?</strong></summary><br/> |
| 335 | |
| 336 | No. Our commercial offerings are priced per node, with volume discounts (smaller price as the number of nodes increases). Our revenue is not related to the number of metrics or the volume of observability data collected or viewed. We designed Netdata for maximum performance at scale and volume for your benefit. Not ours. |
| 337 | |
| 338 | </details> |
| 339 | |
| 340 | <details> |
| 341 | <summary><strong>If I have multiple Parents, how does Netdata Cloud provide unified dashboards?</strong></summary><br/> |
| 342 | |
| 343 | Think of Netdata Cloud as the headend of a distributed database. Each Netdata Parent and Agent dynamically becomes part of that database. So, Netdata Cloud queries them all in parallel, to provide the unified view required. |
| 344 | |
| 345 | </details> |
| 346 | |
| 347 | <details> |
| 348 | <summary><strong>Is querying 100 remote systems in parallel slower than querying a bigger one locally?</strong></summary><br/> |
| 349 | |
| 350 | There is some extra network latency involved, but this is usually small (a few ms), because the data transferred are tiny (your web browser will receive 500-1000 points max, even if the query is 10 days of per-second data). However, the aggregate horse power and parallelism of 100 totally independent systems is orders of magnitude more, compared to any single local system. The queries are actually quite faster. |
| 351 | |
| 352 | </details> |
| 353 | |
| 354 | ## Next Steps |
| 355 | |
| 356 | - **[Deploy your first Agent](/docs/deployment-guides/standalone-deployment.md)** - Start monitoring in 60 seconds |
| 357 | - **[Configure Parents](/docs/deployment-guides/deployment-with-centralization-points.md)** - Scale to hundreds of nodes |
| 358 | - **[Design for Enterprise](/docs/netdata-enterprise-evaluation-corrected.md)** - Architect for thousands |
| 359 | - **[Try Netdata Cloud](/docs/netdata-cloud/README.md)** - Unified visibility across everything |
| 360 | |
| 361 | *Based on real production deployments, independent research (University of Amsterdam, ICSOC 2023), and comparative testing (2025). All metrics and resource usage figures represent typical production scenarios.* |