Add section for scaling parent nodes (#14915)
Update change-metrics-storage.md
Chris Akritidis committed
Apr 17, 2023 at 06:40 UTC
227988441cb0b275b179617990f8bfe82cf322e3
1 file changed
+12
docs/store/change-metrics-storage.md
+12
@@ -127,6 +127,8 @@ The Netdata parent in our production infrastructure at the time of writing:
127
- 3 tiers are used for retention
128
- The `dbengine page cache size MB` in `netdata.conf` is configured to be 4GB
129
130
+Netdata parents can end up collecting millions of metrics per second. See also [scaling dedicated parent nodes](#scaling-dedicated-parent-nodes).
131
+
132
The rule of thumb calculation for this set up gives us
133
```
134
DBENGINE memory = 206,000 x 16 / 1024 MiB = 3,217 MiB = about 3 GiB
@@ -193,3 +195,13 @@ All new child nodes are automatically transferred to the multihost dbengine inst
195
space. If you want to migrate a child node from its legacy dbengine instance to the multihost dbengine instance, you
196
must delete the instance's directory, which is located in `/var/cache/netdata/MACHINE_GUID/dbengine`, after stopping the
197
Agent.
198
+
199
+## Scaling dedicated parent nodes
200
+
201
+When you use streaming in medium to large infrastructures, you can have potentially millions of metrics per second reaching each parent node.
202
+In the lab we have reliably collected 1 million metrics/sec with 16cores and 32GB RAM.
203
+
204
+Our suggestion for scaling parents is to have them running on dedicated VMs, using a maximum of 50% of cpu, and ensuring you have enough RAM
205
+for the desired retention. When your infrastructure can lead a parent to exceed these characteristics, split the load to multiple parents that
206
+do not communicate with each other. With each child sending data to only one of the parents, you can still have replication, high availability,
207
+and infrastructure level observability via the Netdata Cloud UI.