9a8216b96 · cryptotaxi247/netdata-1

@@ -13,11 +13,13 @@ This works. Until it doesn't.

13 When scale breaks the model, teams face two options - and both are wrong:

14

15 ### Option 1: Reduce the Workload

16 +

17 Lower granularity. Drop cardinality. Filter. Sample.

18

19 **This is a trap and a paradox.** If you knew which data you'd need during a crisis, you could predict the crisis and prevent it. By definition, an unpredictable event is the one that will be invisible in your downsampled dataset. You're betting your incident response on being able to predict the unpredictable.

20

21 ### Option 2: Scale the Database

22 +

23 Build giant, expensive clusters. Add more cores. More RAM. More everything.

24

25 **This is a money pit.** In many organizations today, observability costs more than the services being monitored. We routinely encounter companies where 40-50% of infrastructure budget goes to monitoring pipelines, plus teams of specialists to keep them alive.

@@ -27,6 +29,7 @@ Build giant, expensive clusters. Add more cores. More RAM. More everything.

29 **Instead of centralizing data, distribute the code.** This is the heart of Netdata's philosophy and design.

30

31 Every Netdata Agent is a full observability engine:

32 +

33 - Collects metrics at the edge

34 - Stores data locally in multi-tier storage

35 - Runs ML-based anomaly detection in real-time

@@ -46,6 +49,7 @@ Once you pass ~500 nodes, you’re naturally in the multi-million metrics/s rang

49 ## Proof: The Numbers Don't Lie

50

51 ### Independent Validation: University of Amsterdam Study (2023)

52 +

53 **Study:** "An Empirical Evaluation of the Energy and Performance Overhead of Monitoring Tools on Docker-Based Systems"

54 **Conference:** ICSOC 2023 (International Conference on Service-Oriented Computing)

55 **DOI:** 10.1007/978-3-031-48421-6_13

@@ -53,10 +57,11 @@ Once you pass ~500 nodes, you’re naturally in the multi-million metrics/s rang

57 **Finding:** **Netdata is the most energy-efficient monitoring solution**, with the lowest CPU and memory overhead - even while collecting data every second and running anomaly detection at the edge.

58

59 ### Head-to-Head: Netdata vs Prometheus (2025)

60 +

61 We tested a single installation Netdata Parent and Prometheus, at **4.6 million metrics per second** - the scale you hit with just 1,000 nodes. This is how the systems compare for ingestion:

62

59 -| ------------------------------- | ----------------- | ---------------- | -------------------------------------------- |

64 +|---------------------------------|-------------------|------------------|----------------------------------------------|

66 | **Memory Usage** | ~47 GiB | ~383 GiB | 88% less RAM |

@@ -71,7 +76,7 @@ We tested a single installation Netdata Parent and Prometheus, at **4.6 million

76 ### Core Components

77

74 -| ---------- | --------------------------------- | --------------------------------------- | -------------------------------------- | ------------------------ |

79 +|------------|-----------------------------------|-----------------------------------------|----------------------------------------|--------------------------|

@@ -85,6 +90,7 @@ Agents can offload ML, alerting, dashboards, and retention to Parents - typicall

90 ### The Edge Advantage

91

92 Each Agent is autonomous:

93 +

94 - **Collects** 3,000-20,000 metrics per second per node

95 - **Stores** data in tiered storage (raw + aggregated)

96 - **Detects** anomalies using unsupervised ML

@@ -93,6 +99,7 @@ Each Agent is autonomous:

99 - **Streams** to Parents for aggregation

100

101 This means:

102 +

103 - No data loss if Parent is down (Agents buffer locally)

104 - No performance degradation as you scale (work stays distributed)

105 - No architectural changes from 1 to 100,000 nodes

@@ -104,6 +111,7 @@ This means:

111 Parents aren't just centralization points - they're intelligent workload distributors that can reduce the resource footprint on production systems.

112

113 With Parents, Agents can offload:

114 +

115 - **ML Training** - Parents train models, Agents just collect (50% CPU reduction)

116 - **Health Checks** - Parents run all alerts, Agents focus on collection

117 - **Persistent Storage** - Agents run in RAM-only mode with zero disk I/O

@@ -116,12 +124,14 @@ A fully offloaded Agent uses <2% CPU, <150 MiB RAM, and zero disk I/O - a

124 Netdata's ML models flow with the metrics stream, giving you complete flexibility:

125

126 **Option 1: ML at the Edge (default)**

127 +

128 - Agents train their own models locally

129 - Models stream to Parents along with metrics

130 - Parents receive pre-computed ML results

131 - Best for: Systems with available CPU, need for immediate local anomaly detection

132

133 **Option 2: ML at Parents**

134 +

135 - Agents disable ML training (50% CPU savings)

136 - First Parent trains models for all Agents

137 - Models shared with other Parents in cluster

@@ -132,6 +142,7 @@ The architecture adapts to your needs - train where you have resources, use ever

142 ### When You Need Parents

143

144 **We recommend Parents by default:**

145 +

146 - Future-proof your architecture (same setup works at 10 or 100,000 nodes)

147 - Reduce production system load even at small scale

148 - Provide unified dashboards and centralized alerting

@@ -139,6 +150,7 @@ The architecture adapts to your needs - train where you have resources, use ever

150 - Cost less than the resources they save on production systems

151

152 **Parents are essential when you have:**

153 +

154 - **Ephemeral systems** - Kubernetes pods, auto-scaling VMs that disappear

155 - **Resource constraints** - Systems where every CPU cycle matters

156 - **On-premises requirements** - Multi-node view without Cloud connectivity

@@ -147,7 +159,7 @@ The architecture adapts to your needs - train where you have resources, use ever

159 ### Parent Sizing Guidelines

160

161 | Nodes per Parent | Metrics/sec | Resources |

150 -| :--------------: | :---------: | :------------------- |

162 +|:----------------:|:-----------:|:---------------------|

163 | ~100 nodes | ~0.5M/sec | 5 cores, 20 GiB RAM |

164 | ~250 nodes | ~1M/sec | 10 cores, 40 GiB RAM |

165 | ~500 nodes | ~2M/sec | 20 cores, 80 GiB RAM |

@@ -183,12 +195,14 @@ Parents work together intelligently to eliminate duplicate work:

195 Netdata separates automation from monitoring, letting you optimize both:

196

197 **Agents: Local Automation**

198 +

199 - Keep only alerts that trigger local scripts

200 - Example: "If CPU > 90%, scale this service"

201 - Immediate response, no network dependency

202 - Minimal overhead when selective

203

204 **Parents: Human Monitoring**

205 +

206 - Run comprehensive health checks for all Agents

207 - Send notifications to teams via Cloud or integrations

208 - Correlate issues across multiple systems

@@ -201,7 +215,7 @@ This dual approach means production systems only run automation-critical alerts

215 ### Three-Tier Storage System

216

204 -| ---------- | ---------- | -------------------------- | --------------- | ----------------- |

218 +|------------|------------|----------------------------|-----------------|-------------------|

@@ -217,18 +231,21 @@ All tiers update in parallel - no post-processing or compaction jobs needed.

231 ## Why This Architecture Wins

232

233 ### For Operations Teams

234 +

235 - **No blind spots** during incidents - all data available

236 - **No architectural rewrites** as you scale

237 - **No sampling lottery** - the metric you need is always there

238 - **No specialized skills** required - it just works

239

240 ### For Finance

241 +

242 - **Predictable costs** - linear scaling, no surprises

243 - **Lower TCO** - fewer resources for same visibility

244 - **Energy efficient** - independently validated lowest overhead

245 - **Reduced team size** - less complexity to manage

246

247 ### For Developers

248 +

249 - **Per-second granularity** - see what actually happened

250 - **Real-time anomaly detection** - catch issues immediately

251 - **Local dashboards** - debug without central bottlenecks

@@ -274,7 +291,7 @@ No. Agents alone may be enough. Parents are usually required when you have ephem

291 <details>

292 <summary>How much overhead does Netdata introduce on my systems?</summary>

293

277 -Less than 5% CPU and ~200 MiB RAM per agent in standalone mode. Offloaded agents (streaming to a Parent) drop to <2% CPU and ~150 MiB RAM with zero disk I/O. Netdata is designed to be "polite citizen" to production workloads, so it spreads its workload across time and avoids all kinds of sudden and intense spikes.

294 +Less than 5% CPU and ~200 MiB RAM per agent in standalone mode. Offloaded agents (streaming to a Parent) drop to less than 2% CPU and ~150 MiB RAM with zero disk I/O. Netdata is designed to be "polite citizen" to production workloads, so it spreads its workload across time and avoids all kinds of sudden and intense spikes.

295

296 </details>

297

docs: fix mdx parsing scalability.md (#21032)