@cryptotaxi247 / netdata-1 / commits / bc5ba4f89

Update docs on metric storage (#13327)

This PR - Explains the new tiering mechanism. - Housekeeping docs about Agent's database options. - Updates all the configuration options for the `dbengine`. - Provide a new way for the users to calculate the space they need for their metric storage needs (via a spreadsheet) Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud> Co-authored-by: DShreve2 <david@netdata.cloud>

Tasos Katsoulas committed Jul 14, 2022 at 17:16 UTC bc5ba4f8912a5436a458d5015f11c06bd046c01a
5 files changed +329 -294
daemon/config/README.md
+27 -15
@@ -82,21 +82,33 @@ Please note that your data history will be lost if you have modified `history` p
82
83 ### [db] section options
84
85 -| setting | default | info |
86 -|:----------------------------------:|:----------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
87 -| mode | `dbengine` | `dbengine`: The default for long-term metrics storage with efficient RAM and disk usage. Can be extended with `dbengine page cache size MB` and `dbengine disk space MB`. <br />`save`: Netdata will save its round robin database on exit and load it on startup. <br />`map`: Cache files will be updated in real-time. Not ideal for systems with high load or slow disks (check `man mmap`). <br />`ram`: The round-robin database will be temporary and it will be lost when Netdata exits. <br />`none`: Disables the database at this host, and disables health monitoring entirely, as that requires a database of metrics. |
88 -| retention | `3600` | Used with `mode = save/map/ram/alloc`, not the default `mode = dbengine`. This number reflects the number of entries the `netdata` daemon will by default keep in memory for each chart dimension. Check [Memory Requirements](/database/README.md) for more information. |
89 -| update every | `1` | The frequency in seconds, for data collection. For more information see the [performance guide](/docs/guides/configure/performance.md). |
90 -| dbengine page cache size MB | 32 | Determines the amount of RAM in MiB that is dedicated to caching Netdata metric values. |
91 -| dbengine disk space MB | 256 | Determines the amount of disk space in MiB that is dedicated to storing Netdata metric values and all related metadata describing them. |
92 -| dbengine multihost disk space MB | 256 | Same functionality as `dbengine disk space MB`, but includes support for storing metrics streamed to a parent node by its children. Can be used in single-node environments as well. |
93 -| memory deduplication (ksm) | `yes` | When set to `yes`, Netdata will offer its in-memory round robin database and the dbengine page cache to kernel same page merging (KSM) for deduplication. For more information check [Memory Deduplication - Kernel Same Page Merging - KSM](/database/README.md#ksm) |
94 -| cleanup obsolete charts after secs | `3600` | See [monitoring ephemeral containers](/collectors/cgroups.plugin/README.md#monitoring-ephemeral-containers), also sets the timeout for cleaning up obsolete dimensions |
95 -| gap when lost iterations above | `1` | |
96 -| cleanup orphan hosts after secs | `3600` | How long to wait until automatically removing from the DB a remote Netdata host (child) that is no longer sending data. |
97 -| delete obsolete charts files | `yes` | See [monitoring ephemeral containers](/collectors/cgroups.plugin/README.md#monitoring-ephemeral-containers), also affects the deletion of files for obsolete dimensions |
98 -| delete orphan hosts files | `yes` | Set to `no` to disable non-responsive host removal. |
99 -| enable zero metrics | `no` | Set to `yes` to show charts when all their metrics are zero. |
85 +| setting | default | info |
86 +|:---------------------------------------------:|:----------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
87 +| mode | `dbengine` | `dbengine`: The default for long-term metrics storage with efficient RAM and disk usage. Can be extended with `dbengine page cache size MB` and `dbengine disk space MB`. <br />`save`: Netdata will save its round robin database on exit and load it on startup. <br />`map`: Cache files will be updated in real-time. Not ideal for systems with high load or slow disks (check `man mmap`). <br />`ram`: The round-robin database will be temporary and it will be lost when Netdata exits. <br />`none`: Disables the database at this host, and disables health monitoring entirely, as that requires a database of metrics. |
88 +| retention | `3600` | Used with `mode = save/map/ram/alloc`, not the default `mode = dbengine`. This number reflects the number of entries the `netdata` daemon will by default keep in memory for each chart dimension. Check [Memory Requirements](/database/README.md) for more information. |
89 +| storage tiers | `1` | The number of storage tiers you want to have in your dbengine. Check the tiering mechanism in the [dbengine's reference](/database/engine/README.md#tiering). You can have up to 5 tiers of data (including the _Tier 0_). This number ranges between 1 and 5. |
90 +| dbengine page cache size MB | `32` | Determines the amount of RAM in MiB that is dedicated to caching for _Tier 0_ Netdata metric values. |
91 +| dbengine tier **`N`** page cache size MB | `32` | Determines the amount of RAM in MiB that is dedicated for caching Netdata metric values of the **`N`** tier. <br /> `N belongs to [1..4]` ||
92 + | dbengine disk space MB | `256` | Determines the amount of disk space in MiB that is dedicated to storing _Tier 0_ Netdata metric values and all related metadata describing them. This option is available **only for legacy configuration** (`Agent v1.23.2 and prior`). |
93 +| dbengine multihost disk space MB | `256` | Same functionality as `dbengine disk space MB`, but includes support for storing metrics streamed to a parent node by its children. Can be used in single-node environments as well. This setting is only for _Tier 0_ metrics. |
94 +| dbengine tier **`N`** multihost disk space MB | `256` | Same functionality as `dbengine multihost disk space MB`, but stores metrics of the **`N`** tier (both parent node and its children). Can be used in single-node environments as well. <br /> `N belongs to [1..4]` |
95 +| update every | `1` | The frequency in seconds, for data collection. For more information see the [performance guide](/docs/guides/configure/performance.md). These metrics stored as _Tier 0_ data. Explore the tiering mechanism in the [dbengine's reference](/database/engine/README.md#tiering). |
96 +| dbengine tier **`N`** update every iterations | `60` | The down sampling value of each tier from the previous one. For each Tier, the greater by one Tier has N (equal to 60 by default) less data points of any metric it collects. This setting can take values from `2` up to `255`. <br /> `N belongs to [1..4]` |
97 +| dbengine tier **`N`** back fill | `New` | Specifies the strategy of recreating missing data on each Tier from the exact lower Tier. <br /> `New`: Sees the latest point on each Tier and save new points to it only if the exact lower Tier has available points for it's observation window (`dbengine tier N update every iterations` window). <br /> `none`: No back filling is applied. <br /> `N belongs to [1..4]` |
98 +| memory deduplication (ksm) | `yes` | When set to `yes`, Netdata will offer its in-memory round robin database and the dbengine page cache to kernel same page merging (KSM) for deduplication. For more information check [Memory Deduplication - Kernel Same Page Merging - KSM](/database/README.md#ksm) |
99 +| cleanup obsolete charts after secs | `3600` | See [monitoring ephemeral containers](/collectors/cgroups.plugin/README.md#monitoring-ephemeral-containers), also sets the timeout for cleaning up obsolete dimensions |
100 +| gap when lost iterations above | `1` | |
101 +| cleanup orphan hosts after secs | `3600` | How long to wait until automatically removing from the DB a remote Netdata host (child) that is no longer sending data. |
102 +| delete obsolete charts files | `yes` | See [monitoring ephemeral containers](/collectors/cgroups.plugin/README.md#monitoring-ephemeral-containers), also affects the deletion of files for obsolete dimensions |
103 +| delete orphan hosts files | `yes` | Set to `no` to disable non-responsive host removal. |
104 +| enable zero metrics | `no` | Set to `yes` to show charts when all their metrics are zero. |
105 +
106 +:::info
107 +
108 +The multiplication of all the **enabled** tiers `dbengine tier N update every iterations` values must be less than `65535`.
109 +
110 +:::
111 +
112
113 ### [directories] section options
114
database/README.md
+76 -137
@@ -7,199 +7,138 @@ custom_edit_url: https://github.com/netdata/netdata/edit/master/database/README.
7 # Database
8
9 Netdata is fully capable of long-term metrics storage, at per-second granularity, via its default database engine
10 -(`dbengine`). But to remain as flexible as possible, Netdata supports a number of types of metrics storage:
10 +(`dbengine`). But to remain as flexible as possible, Netdata supports several storage options:
11
12 1. `dbengine`, (the default) data are in database files. The [Database Engine](/database/engine/README.md) works like a
13 - traditional database. There is some amount of RAM dedicated to data caching and indexing and the rest of the data
14 - reside compressed on disk. The number of history entries is not fixed in this case, but depends on the configured
15 - disk space and the effective compression ratio of the data stored. This is the **only mode** that supports changing
16 - the data collection update frequency (`update every`) **without losing** the previously stored metrics. For more
17 - details see [here](/database/engine/README.md).
13 + traditional database. There is some amount of RAM dedicated to data caching and indexing and the rest of the data
14 + reside compressed on disk. The number of history entries is not fixed in this case, but depends on the configured
15 + disk space and the effective compression ratio of the data stored. This is the **only mode** that supports changing
16 + the data collection update frequency (`update every`) **without losing** the previously stored metrics. For more
17 + details see [here](/database/engine/README.md).
18
19 -2. `ram`, data are purely in memory. Data are never saved on disk. This mode uses `mmap()` and supports [KSM](#ksm).
19 +2. `ram`, data are purely in memory. Data are never saved on disk. This mode uses `mmap()` and supports [KSM](#ksm).
20
21 -3. `save`, data are only in RAM while Netdata runs and are saved to / loaded from disk on Netdata
22 - restart. It also uses `mmap()` and supports [KSM](#ksm).
21 +3. `save`, data are only in RAM while Netdata runs and are saved to / loaded from disk on Netdata restart. It also
22 + uses `mmap()` and supports [KSM](#ksm).
23
24 -4. `map`, data are in memory mapped files. This works like the swap. Keep in mind though, this will have a constant
25 - write on your disk. When Netdata writes data on its memory, the Linux kernel marks the related memory pages as dirty
26 - and automatically starts updating them on disk. Unfortunately we cannot control how frequently this works. The Linux
27 - kernel uses exactly the same algorithm it uses for its swap memory. Check below for additional information on
28 - running a dedicated central Netdata server. This mode uses `mmap()` but does not support [KSM](#ksm).
24 +4. `map`, data are in memory mapped files. This works like the swap. When Netdata writes data on its memory, the Linux
25 + kernel marks the related memory pages as dirty and automatically starts updating them on disk. Unfortunately we
26 + cannot control how frequently this works. The Linux kernel uses exactly the same algorithm it uses for its swap
27 + memory. This mode uses `mmap()` but does not support [KSM](#ksm). _Keep in mind though, this option will have a
28 + constant write on your disk._
29
30 -5. `none`, without a database (collected metrics can only be streamed to another Netdata).
30 +5. `alloc`, like `ram` but it uses `calloc()` and does not support [KSM](#ksm). This mode is the fallback for all others
31 + except `none`.
32
32 -6. `alloc`, like `ram` but it uses `calloc()` and does not support [KSM](#ksm). This mode is the fallback for all
33 - others except `none`.
33 +6. `none`, without a database (collected metrics can only be streamed to another Netdata).
34
35 -You can select the database mode by editing `netdata.conf` and setting:
36 -
37 -```conf
38 -[db]
39 - # dbengine (default), ram, save (the default if dbengine not available), map (swap like), none, alloc
40 - mode = dbengine
41 -```
42 -
43 -## Running Netdata in embedded devices
44 -
45 -Embedded devices usually have very limited RAM resources available.
46 -
47 -There are 2 settings for you to tweak:
48 -
49 -1. `[db].update every`, which controls the data collection frequency
50 -2. `[db].retention`, which controls the size of the database in memory (except for `[db].mode = dbengine`)
51 -
52 -By default `[db].update every = 1` and `[db].retention = 3600`. This gives you an hour of data with per second updates.
53 -
54 -If you set `[db].update every = 2` and `[db].retention = 1800`, you will still have an hour of data, but collected once every 2
55 -seconds. This will **cut in half** both CPU and RAM resources consumed by Netdata. Of course experiment a bit. On very
56 -weak devices you might have to use `[db].update every = 5` and `[db].retention = 720` (still 1 hour of data, but 1/5 of the CPU and
57 -RAM resources).
58 -
59 -You can also disable [data collection plugins](/collectors/README.md) you don't need. Disabling such plugins will also free both
60 -CPU and RAM resources.
61 -
62 -## Running a dedicated parent Netdata server
63 -
64 -Netdata allows streaming data between Netdata nodes in real-time. This allows having one or more parent Netdata servers that will maintain
65 -the entire database for all the nodes that connect to them (their children), and will also run health checks/alarms for all these nodes.
66 -
67 -### map
35 +## Which database mode to use
36
69 -In this mode, the database of Netdata is stored in memory mapped files. Netdata continues to read and write the database
70 -in memory, but the kernel automatically loads and saves memory pages from/to disk.
37 +The default mode `[db].mode = dbengine` has been designed to scale for longer retentions and is the only mode suitable
38 +for parent Agents in the _Parent - Child_ setups
39
72 -**We suggest _not_ to use this mode on nodes that run other applications.** There will always be dirty memory to be
73 -synced and this syncing process may influence the way other applications work. This mode however is useful when we need
74 -a parent Netdata server that would normally need huge amounts of memory.
40 +The other available database modes are designed to minimize resource utilization and should only be considered on
41 +[Parent - Child](/docs/metrics-storage-management/how-streaming-works) setups at the children side and only when the
42 +resource constraints are very strict.
43
76 -There are a few kernel options that provide finer control on the way this syncing works. But before explaining them, a
77 -brief introduction of how Netdata database works is needed.
44 +So,
45
79 -For each chart, Netdata maps the following files:
46 +- On a single node setup, use `[db].mode = dbengine`.
47 +- On a [Parent - Child](/docs/metrics-storage-management/how-streaming-works) setup, use `[db].mode = dbengine` on the
48 + parent to increase retention, a more resource efficient mode like, `dbengine` with light retention settings, and
49 + `save`, `ram` or `none` modes for the children to minimize resource utilization.
50
81 -1. `chart/main.db`, this is the file that maintains chart information. Every time data are collected for a chart, this
82 - is updated.
83 -2. `chart/dimension_name.db`, this is the file for each dimension. At its beginning there is a header, followed by the
84 - round robin database where metrics are stored.
51 +## Choose your database mode
52
86 -So, every time Netdata collects data, the following pages will become dirty:
87 -
88 -1. the chart file
89 -2. the header part of all dimension files
90 -3. if the collected metrics are stored far enough in the dimension file, another page will become dirty, for each
91 - dimension
92 -
93 -Each page in Linux is 4KB. So, with 200 charts and 1000 dimensions, there will be 1200 to 2200 4KB pages dirty pages
94 -every second. Of course 1200 of them will always be dirty (the chart header and the dimensions headers) and 1000 will be
95 -dirty for about 1000 seconds (4 bytes per metric, 4KB per page, so 1000 seconds, or 16 minutes per page).
96 -
97 -Hopefully, the Linux kernel does not sync all these data every second. The frequency they are synced is controlled by
98 -`/proc/sys/vm/dirty_expire_centisecs` or the `sysctl` `vm.dirty_expire_centisecs`. The default on most systems is 3000
99 -(30 seconds).
100 -
101 -On a busy server centralizing metrics from 20+ servers you will experience this:
102 -
103 -![image](https://cloud.githubusercontent.com/assets/2662304/23834750/429ab0dc-0764-11e7-821a-d7908bc881ac.png)
104 -
105 -As you can see, there is quite some stress (this is `iowait`) every 30 seconds.
106 -
107 -A simple solution is to increase this time to 10 minutes (60000). This is the same system with this setting in 10
108 -minutes:
109 -
110 -![image](https://cloud.githubusercontent.com/assets/2662304/23834784/d2304f72-0764-11e7-8389-fb830ffd973a.png)
111 -
112 -Of course, setting this to 10 minutes means that data on disk might be up to 10 minutes old if you get an abnormal
113 -shutdown.
53 +You can select the database mode by editing `netdata.conf` and setting:
54
115 -There are 2 more options to tweak:
55 +```conf
56 +[db]
57 + # dbengine (default), ram, save (the default if dbengine not available), map (swap like), none, alloc
58 + mode = dbengine
59 +```
60
117 -1. `dirty_background_ratio`, by default `10`.
118 -2. `dirty_ratio`, by default `20`.
61 +## Netdata Longer Metrics Retention
62
120 -These control the amount of memory that should be dirty for disk syncing to be triggered. On dedicated Netdata servers,
121 -you can use: `80` and `90` respectively, so that all RAM is given to Netdata.
63 +Metrics retention is controlled only by the disk space allocated to storing metrics. But it also affects the memory and
64 +CPU required by the agent to query longer timeframes.
65
123 -With these settings, you can expect a little `iowait` spike once every 10 minutes and in case of system crash, data on
124 -disk will be up to 10 minutes old.
66 +Since Netdata Agents usually run on the edge, on production systems, Netdata Agent **parents** should be considered.
67 +When having a [**parent - child**](/docs/metrics-storage-management/how-streaming-works.md) setup, the child (the
68 +Netdata Agent running on a production system) delegates all of its functions, including longer metrics retention and
69 +querying, to the parent node that can dedicate more resources to this task. A single Netdata Agent parent can centralize
70 +multiple children Netdata Agents (dozens, hundreds, or even thousands depending on its available resources).
71
126 -![image](https://cloud.githubusercontent.com/assets/2662304/23835030/ba4bf506-0768-11e7-9bc6-3b23e080c69f.png)
72 +## Running Netdata on embedded devices
73
128 -To have these settings automatically applied on boot, create the file `/etc/sysctl.d/netdata-memory.conf` with these
129 -contents:
74 +Embedded devices typically have very limited RAM resources available.
75
131 -```conf
132 -vm.dirty_expire_centisecs = 60000
133 -vm.dirty_background_ratio = 80
134 -vm.dirty_ratio = 90
135 -vm.dirty_writeback_centisecs = 0
136 -```
76 +There are two settings for you to configure:
77
138 -There is another mode to help overcome the memory size problem. What is **most interesting for this setup** is
139 -`[db].mode = dbengine`.
78 +1. `[db].update every`, which controls the data collection frequency
79 +2. `[db].retention`, which controls the size of the database in memory (except for `[db].mode = dbengine`)
80
141 -### dbengine
81 +By default `[db].update every = 1` and `[db].retention = 3600`. This gives you an hour of data with per second updates.
82
143 -In this mode, the database of Netdata is stored in database files. The [Database Engine](/database/engine/README.md)
144 -works like a traditional database. There is some amount of RAM dedicated to data caching and indexing and the rest of
145 -the data reside compressed on disk. The number of history entries is not fixed in this case, but depends on the
146 -configured disk space and the effective compression ratio of the data stored.
83 +If you set `[db].update every = 2` and `[db].retention = 1800`, you will still have an hour of data, but collected once
84 +every 2 seconds. This will **cut in half** both CPU and RAM resources consumed by Netdata. Of course experiment a bit to find the right setting.
85 +On very weak devices you might have to use `[db].update every = 5` and `[db].retention = 720` (still 1 hour of data, but
86 +1/5 of the CPU and RAM resources).
87
148 -We suggest to use **this** mode on nodes that also run other applications. The Database Engine uses direct I/O to avoid
149 -polluting the OS filesystem caches and does not generate excessive I/O traffic so as to create the minimum possible
150 -interference with other applications. Using mode `dbengine` we can overcome most memory restrictions. For more
151 -details see [here](/database/engine/README.md).
88 +You can also disable [data collection plugins](/collectors/README.md) that you don't need. Disabling such plugins will also
89 +free both CPU and RAM resources.
90
153 -## KSM
91 +## Memory optimizations
92
155 -Netdata offers all its in-memory database to kernel for deduplication.
93 +### KSM
94
157 -In the past KSM has been criticized for consuming a lot of CPU resources. Although this is true when KSM is used for
158 -deduplicating certain applications, it is not true with netdata, since the Netdata memory is written very infrequently
159 -(if you have 24 hours of metrics in netdata, each byte at the in-memory database will be updated just once per day).
95 +KSM performs memory deduplication by scanning through main memory for physical pages that have identical content, and
96 +identifies the virtual pages that are mapped to those physical pages. It leaves one page unchanged, and re-maps each
97 +duplicate page to point to the same physical page. Netdata offers all of its in-memory database to kernel for
98 +deduplication.
99
161 -KSM is a solution that will provide 60+% memory savings to Netdata.
100 +In the past, KSM has been criticized for consuming a lot of CPU resources. This is true when KSM is used for
101 +deduplicating certain applications, but it is not true for Netdata. Agent's memory is written very infrequently
102 +(if you have 24 hours of metrics in Netdata, each byte at the in-memory database will be updated just once per day). KSM
103 +is a solution that will provide 60+% memory savings to Netdata.
104
105 ### Enable KSM in kernel
106
165 -You need to run a kernel compiled with:
107 +To enable KSM in kernel, you need to run a kernel compiled with the following:
108
109 ```sh
110 CONFIG_KSM=y
111 ```
112
171 -When KSM is enabled at the kernel is just available for the user to enable it.
113 +When KSM is enabled at the kernel, it is just available for the user to enable it.
114
173 -So, if you build a kernel with `CONFIG_KSM=y` you will just get a few files in `/sys/kernel/mm/ksm`. Nothing else
174 -happens. There is no performance penalty (apart I guess from the memory this code occupies into the kernel).
115 +If you build a kernel with `CONFIG_KSM=y`, you will just get a few files in `/sys/kernel/mm/ksm`. Nothing else
116 +happens. There is no performance penalty (apart from the memory this code occupies into the kernel).
117
118 The files that `CONFIG_KSM=y` offers include:
119
178 -- `/sys/kernel/mm/ksm/run` by default `0`. You have to set this to `1` for the
179 - kernel to spawn `ksmd`.
180 -- `/sys/kernel/mm/ksm/sleep_millisecs`, by default `20`. The frequency ksmd
181 - should evaluate memory for deduplication.
182 -- `/sys/kernel/mm/ksm/pages_to_scan`, by default `100`. The amount of pages
183 - ksmd will evaluate on each run.
120 +- `/sys/kernel/mm/ksm/run` by default `0`. You have to set this to `1` for the kernel to spawn `ksmd`.
121 +- `/sys/kernel/mm/ksm/sleep_millisecs`, by default `20`. The frequency ksmd should evaluate memory for deduplication.
122 +- `/sys/kernel/mm/ksm/pages_to_scan`, by default `100`. The amount of pages ksmd will evaluate on each run.
123
124 So, by default `ksmd` is just disabled. It will not harm performance and the user/admin can control the CPU resources
186 -he/she is willing `ksmd` to use.
125 +they are willing to have used by `ksmd`.
126
127 ### Run `ksmd` kernel daemon
128
190 -To activate / run `ksmd` you need to run:
129 +To activate / run `ksmd,` you need to run the following:
130
131 ```sh
132 echo 1 >/sys/kernel/mm/ksm/run
133 echo 1000 >/sys/kernel/mm/ksm/sleep_millisecs
134 ```
135
197 -With these settings ksmd does not even appear in the running process list (it will run once per second and evaluate 100
136 +With these settings, ksmd does not even appear in the running process list (it will run once per second and evaluate 100
137 pages for de-duplication).
138
139 Put the above lines in your boot sequence (`/etc/rc.local` or equivalent) to have `ksmd` run at boot.
140
202 -## Monitoring Kernel Memory de-duplication performance
141 +### Monitoring Kernel Memory de-duplication performance
142
143 Netdata will create charts for kernel memory de-duplication performance, like this:
144
database/engine/README.md
+125 -81
@@ -6,74 +6,114 @@ custom_edit_url: https://github.com/netdata/netdata/edit/master/database/engine/
6
7 # Database engine
8
9 -The Database Engine works like a traditional database. It dedicates a certain amount of RAM to data caching and
10 -indexing, while the rest of the data resides compressed on disk. Unlike other [database modes](/database/README.md), the
11 -amount of historical metrics stored is based on the amount of disk space you allocate and the effective compression
9 +The Database Engine works like a traditional time series database. Unlike other [database modes](/database/README.md),
10 +the amount of historical metrics stored is based on the amount of disk space you allocate and the effective compression
11 ratio, not a fixed number of metrics collected.
12
14 -By using both RAM and disk space, the database engine allows for long-term storage of per-second metrics inside of the
15 -Agent itself.
13 +## Tiering
14
17 -In addition, the dbengine is the only mode that supports changing the data collection update frequency
18 -(`update every`) without losing the metrics your Agent already gathered and stored.
15 +Tiering is a mechanism of providing multiple tiers of data with
16 +different [granularity on metrics](/docs/store/distributed-data-architecture.md#granularity-of-metrics).
17
20 -## Configuration
18 +For Netdata Agents with version `netdata-1.35.0.138.nightly` and greater, `dbengine` supports Tiering, allowing almost
19 +unlimited retention of data.
20
22 -To use the database engine, open `netdata.conf` and set `[db].mode` to `dbengine`.
21
24 -```conf
22 +### Metric size
23 +
24 +Every Tier down samples the exact lower tier (lower tiers have greater resolution). You can have up to 5
25 +Tiers **[0. . 4]** of data (including the Tier 0, which has the highest resolution)
26 +
27 +Tier 0 is the default that was always available in `dbengine` mode. Tier 1 is the first level of aggregation, Tier 2 is
28 +the second, and so on.
29 +
30 +Metrics on all tiers except of the _Tier 0_ also store the following five additional values for every point for accurate
31 +representation:
32 +
33 +1. The `sum` of the points aggregated
34 +2. The `min` of the points aggregated
35 +3. The `max` of the points aggregated
36 +4. The `count` of the points aggregated (could be constant, but it may not be due to gaps in data collection)
37 +5. The `anomaly_count` of the points aggregated (how many of the aggregated points found anomalous)
38 +
39 +Among `min`, `max` and `sum`, the correct value is chosen based on the user query. `average` is calculated on the fly at
40 +query time.
41 +
42 +### Tiering in a nutshell
43 +
44 +The `dbengine` is capable of retaining metrics for years. To further understand the `dbengine` tiering mechanism let's
45 +explore the following configuration.
46 +
47 +```
48 [db]
49 mode = dbengine
50 +
51 + # per second data collection
52 + update every = 1
53 +
54 + # enables Tier 1 and Tier 2, Tier 0 is always enabled in dbengine mode
55 + storage tiers = 3
56 +
57 + # Tier 0, per second data for a week
58 + dbengine multihost disk space MB = 1100
59 +
60 + # Tier 1, per minute data for a month
61 + dbengine tier 1 multihost disk space MB = 330
62 +
63 + # Tier 2, per hour data for a year
64 + dbengine tier 2 multihost disk space MB = 67
65 ```
66
29 -To configure the database engine, look for the `dbengine page cache size MB` and `dbengine multihost disk space MB` settings in the
30 -`[db]` section of your `netdata.conf`. The Agent ignores the `[db].retention` setting when using the dbengine.
67 +For 2000 metrics, collected every second and retained for a week, Tier 0 needs: 1 byte x 2000 metrics x 3600 secs per
68 +hour x 24 hours per day x 7 days per week = 1100MB.
69
32 -```conf
33 -[db]
34 - dbengine page cache size MB = 32
35 - dbengine multihost disk space MB = 256
36 -```
70 +By setting `dbengine multihost disk space MB` to `1100`, this node will start maintaining about a week of data. But pay
71 +attention to the number of metrics. If you have more than 2000 metrics on a node, or you need more that a week of high
72 +resolution metrics, you may need to adjust this setting accordingly.
73 +
74 +Tier 1 is by default sampling the data every **60 points of Tier 0**. In our case, Tier 0 is per second, if we want to
75 +transform this information in terms of time then the Tier 1 "resolution" is per minute.
76 +
77 +Tier 1 needs four times more storage per point compared to Tier 0. So, for 2000 metrics, with per minute resolution,
78 +retained for a month, Tier 1 needs: 4 bytes x 2000 metrics x 60 minutes per hour x 24 hours per day x 30 days per month
79 += 330MB.
80 +
81 +Tier 2 is by default sampling data every 3600 points of Tier 0 (60 of Tier 1, which is the previous exact Tier). Again
82 +in term of "time" (Tier 0 is per second), then Tier 2 is per hour.
83 +
84 +The storage requirements are the same to Tier 1.
85
38 -The above values are the default values for Page Cache size and DB engine disk space quota.
86 +For 2000 metrics, with per hour resolution, retained for a year, Tier 2 needs: 4 bytes x 2000 metrics x 24 hours per day
87 +x 365 days per year = 67MB.
88
40 -The `dbengine page cache size MB` option determines the amount of RAM dedicated to caching Netdata metric values. The
41 -actual page cache size will be slightly larger than this figure—see the [memory requirements](#memory-requirements)
42 -section for details.
89 +## Legacy configuration
90
44 -The `dbengine multihost disk space MB` option determines the amount of disk space that is dedicated to storing
45 -Netdata metric values and all related metadata describing them. You can use the [**database engine
46 -calculator**](/docs/store/change-metrics-storage.md#calculate-the-system-resources-ram-disk-space-needed-to-store-metrics)
47 -to correctly set `dbengine multihost disk space MB` based on your metrics retention policy. The calculator gives an
48 -accurate estimate based on how many child nodes you have, how many metrics your Agent collects, and more.
91 +### v1.35.1 and prior
92
50 -### Legacy configuration
93 +These versions of the Agent do not support [Tiering](#Tiering). You could change the metric retention for the parent and
94 +all of its children only with the `dbengine multihost disk space MB` setting. This setting accounts the space allocation
95 +for the parent node and all of its children.
96
52 -The deprecated `dbengine disk space MB` option determines the amount of disk space that is dedicated to storing
53 -Netdata metric values per legacy database engine instance (see [details on the legacy mode](#legacy-mode) below).
97 +To configure the database engine, look for the `page cache size MB` and `dbengine multihost disk space MB` settings in
98 +the `[db]` section of your `netdata.conf`.
99
100 ```conf
101 [db]
57 - dbengine disk space MB = 256
102 + dbengine page cache size MB = 32
103 + dbengine multihost disk space MB = 256
104 ```
105
60 -### Streaming metrics to the database engine
61 -
62 -When using the multihost database engine, all parent and child nodes share the same `dbengine page cache size MB` and `dbengine
63 -multihost disk space MB` in a single dbengine instance. The [**database engine
64 -calculator**](/docs/store/change-metrics-storage.md#calculate-the-system-resources-ram-disk-space-needed-to-store-metrics)
65 -helps you properly set `dbengine page cache size MB` and `dbengine multihost disk space MB` on your parent node to allocate enough
66 -resources based on your metrics retention policy and how many child nodes you have.
67 -
68 -#### Legacy mode
106 +### v1.23.2 and prior
107
108 _For Netdata Agents earlier than v1.23.2_, the Agent on the parent node uses one dbengine instance for itself, and
109 another instance for every child node it receives metrics from. If you had four streaming nodes, you would have five
110 instances in total (`1 parent + 4 child nodes = 5 instances`).
111
74 -The Agent allocates resources for each instance separately using the `dbengine disk space MB` (**deprecated**) setting. If
75 -`dbengine disk space MB`(**deprecated**) is set to the default `256`, each instance is given 256 MiB in disk space, which
76 -means the total disk space required to store all instances is, roughly, `256 MiB * 1 parent * 4 child nodes = 1280 MiB`.
112 +The Agent allocates resources for each instance separately using the `dbengine disk space MB` (**deprecated**) setting.
113 +If
114 +`dbengine disk space MB`(**deprecated**) is set to the default `256`, each instance is given 256 MiB in disk space,
115 +which means the total disk space required to store all instances is,
116 +roughly, `256 MiB * 1 parent * 4 child nodes = 1280 MiB`.
117
118 #### Backward compatibility
119
@@ -90,41 +130,44 @@ Agent.
130 For more information about setting `[db].mode` on your nodes, in addition to other streaming configurations, see
131 [streaming](/streaming/README.md).
132
93 -### Memory requirements
133 +## Requirements & limitations
134 +
135 +### Memory
136
137 Using database mode `dbengine` we can overcome most memory restrictions and store a dataset that is much larger than the
138 available memory.
139
140 There are explicit memory requirements **per** DB engine **instance**:
141
100 -- The total page cache memory footprint will be an additional `#dimensions-being-collected x 4096 x 2` bytes over what
101 - the user configured with `dbengine page cache size MB`.
142 +- The total page cache memory footprint will be an additional `#dimensions-being-collected x 4096 x 2` bytes over what
143 + the user configured with `dbengine page cache size MB`.
144 +
145
103 -- an additional `#pages-on-disk x 4096 x 0.03` bytes of RAM are allocated for metadata.
146 +- an additional `#pages-on-disk x 4096 x 0.03` bytes of RAM are allocated for metadata.
147
105 - - roughly speaking this is 3% of the uncompressed disk space taken by the DB files.
148 + - roughly speaking this is 3% of the uncompressed disk space taken by the DB files.
149
107 - - for very highly compressible data (compression ratio > 90%) this RAM overhead is comparable to the disk space
108 - footprint.
150 + - for very highly compressible data (compression ratio > 90%) this RAM overhead is comparable to the disk space
151 + footprint.
152
153 An important observation is that RAM usage depends on both the `page cache size` and the `dbengine multihost disk space`
154 options.
155
113 -You can use our [database engine
114 -calculator](/docs/store/change-metrics-storage.md#calculate-the-system-resources-ram-disk-space-needed-to-store-metrics)
156 +You can use
157 +our [database engine calculator](/docs/store/change-metrics-storage.md#calculate-the-system-resources-ram-disk-space-needed-to-store-metrics)
158 to validate the memory requirements for your particular system(s) and configuration (**out-of-date**).
159
117 -### Disk space requirements
160 +### Disk space
161
162 There are explicit disk space requirements **per** DB engine **instance**:
163
121 -- The total disk space footprint will be the maximum between `#dimensions-being-collected x 4096 x 2` bytes or what
122 - the user configured with `dbengine multihost disk space` or `dbengine disk space`.
164 +- The total disk space footprint will be the maximum between `#dimensions-being-collected x 4096 x 2` bytes or what the
165 + user configured with `dbengine multihost disk space` or `dbengine disk space`.
166
124 -### File descriptor requirements
167 +### File descriptor
168
126 -The Database Engine may keep a **significant** amount of files open per instance (e.g. per streaming child or
127 -parent server). When configuring your system you should make sure there are at least 50 file descriptors available per
169 +The Database Engine may keep a **significant** amount of files open per instance (e.g. per streaming child or parent
170 +server). When configuring your system you should make sure there are at least 50 file descriptors available per
171 `dbengine` instance.
172
173 Netdata allocates 25% of the available file descriptors to its Database Engine instances. This means that only 25% of
@@ -148,7 +191,7 @@ ulimit -n 65536
191 ```
192
193 at the beginning of the service file. Alternatively you can change the system-wide limits of the kernel by changing
151 - `/etc/sysctl.conf`. For linux that would be:
194 +`/etc/sysctl.conf`. For linux that would be:
195
196 ```conf
197 fs.file-max = 65536
@@ -165,8 +208,8 @@ You can apply the settings by running `sysctl -p` or by rebooting.
208
209 ## Files
210
168 -With the DB engine mode the metric data are stored in database files. These files are organized in pairs, the
169 -datafiles and their corresponding journalfiles, e.g.:
211 +With the DB engine mode the metric data are stored in database files. These files are organized in pairs, the datafiles
212 +and their corresponding journalfiles, e.g.:
213
214 ```sh
215 datafile-1-0000000001.ndf
@@ -191,15 +234,16 @@ storage at lower granularity.
234 The DB engine stores chart metric values in 4096-byte pages in memory. Each chart dimension gets its own page to store
235 consecutive values generated from the data collectors. Those pages comprise the **Page Cache**.
236
194 -When those pages fill up they are slowly compressed and flushed to disk. It can take `4096 / 4 = 1024 seconds = 17
195 -minutes`, for a chart dimension that is being collected every 1 second, to fill a page. Pages can be cut short when we
196 -stop Netdata or the DB engine instance so as to not lose the data. When we query the DB engine for data we trigger disk
197 -read I/O requests that fill the Page Cache with the requested pages and potentially evict cold (not recently used)
198 -pages.
237 +When those pages fill up, they are slowly compressed and flushed to disk. It can
238 +take `4096 / 4 = 1024 seconds = 17 minutes`, for a chart dimension that is being collected every 1 second, to fill a
239 +page. Pages can be cut short when we stop Netdata or the DB engine instance so as to not lose the data. When we query
240 +the DB engine for data we trigger disk read I/O requests that fill the Page Cache with the requested pages and
241 +potentially evict cold (not recently used)
242 +pages.
243
244 When the disk quota is exceeded the oldest values are removed from the DB engine at real time, by automatically deleting
245 the oldest datafile and journalfile pair. Any corresponding pages residing in the Page Cache will also be invalidated
202 -and removed. The DB engine logic will try to maintain between 10 and 20 file pairs at any point in time.
246 +and removed. The DB engine logic will try to maintain between 10 and 20 file pairs at any point in time.
247
248 The Database Engine uses direct I/O to avoid polluting the OS filesystem caches and does not generate excessive I/O
249 traffic so as to create the minimum possible interference with other applications.
@@ -214,19 +258,19 @@ Constellation ES.3 2TB magnetic HDD and a SAMSUNG MZQLB960HAJR-00007 960GB NAND
258 For our workload, we defined 32 charts with 128 metrics each, giving us a total of 4096 metrics. We defined 1 worker
259 thread per chart (32 threads) that generates new data points with a data generation interval of 1 second. The time axis
260 of the time-series is emulated and accelerated so that the worker threads can generate as many data points as possible
217 -without delays.
261 +without delays.
262
219 -We also defined 32 worker threads that perform queries on random metrics with semi-random time ranges. The
220 -starting time of the query is randomly selected between the beginning of the time-series and the time of the latest data
221 -point. The ending time is randomly selected between 1 second and 1 hour after the starting time. The pseudo-random
222 -numbers are generated with a uniform distribution.
263 +We also defined 32 worker threads that perform queries on random metrics with semi-random time ranges. The starting time
264 +of the query is randomly selected between the beginning of the time-series and the time of the latest data point. The
265 +ending time is randomly selected between 1 second and 1 hour after the starting time. The pseudo-random numbers are
266 +generated with a uniform distribution.
267
268 The data are written to the database at the same time as they are read from it. This is a concurrent read/write mixed
225 -workload with a duration of 60 seconds. The faster `dbengine` runs, the bigger the dataset size becomes since more
226 -data points will be generated. We set a page cache size of 64MiB for the two disk-bound scenarios. This way, the dataset
227 -size of the metric data is much bigger than the RAM that is being used for caching so as to trigger I/O requests most
228 -of the time. In our final scenario, we set the page cache size to 16 GiB. That way, the dataset fits in the page cache
229 -so as to avoid all disk bottlenecks.
269 +workload with a duration of 60 seconds. The faster `dbengine` runs, the bigger the dataset size becomes since more data
270 +points will be generated. We set a page cache size of 64MiB for the two disk-bound scenarios. This way, the dataset size
271 +of the metric data is much bigger than the RAM that is being used for caching so as to trigger I/O requests most of the
272 +time. In our final scenario, we set the page cache size to 16 GiB. That way, the dataset fits in the page cache so as to
273 +avoid all disk bottlenecks.
274
275 The reported numbers are the following:
276
@@ -237,15 +281,15 @@ The reported numbers are the following:
281 | N/A | 16 GiB | 6.8 GiB | 118.2M | 30.2M |
282
283 where "reads/sec" is the number of metric data points being read from the database via its API per second and
240 -"writes/sec" is the number of metric data points being written to the database per second.
284 +"writes/sec" is the number of metric data points being written to the database per second.
285
286 Notice that the HDD numbers are pretty high and not much slower than the SSD numbers. This is thanks to the database
287 engine design being optimized for rotating media. In the database engine disk I/O requests are:
288
245 -- asynchronous to mask the high I/O latency of HDDs.
246 -- mostly large to reduce the amount of HDD seeking time.
247 -- mostly sequential to reduce the amount of HDD seeking time.
248 -- compressed to reduce the amount of required throughput.
289 +- asynchronous to mask the high I/O latency of HDDs.
290 +- mostly large to reduce the amount of HDD seeking time.
291 +- mostly sequential to reduce the amount of HDD seeking time.
292 +- compressed to reduce the amount of required throughput.
293
294 As a result, the HDD is not thousands of times slower than the SSD, which is typical for other workloads.
295
docs/store/change-metrics-storage.md
+59 -36
@@ -6,72 +6,95 @@ custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/store/chang
6
7 # Change how long Netdata stores metrics
8
9 -import { Calculator } from '../../src/components/agent/dbCalc/'
9 +The Netdata Agent uses a custom made time-series database (TSDB), named the [`dbengine`](/database/engine/README.md), to store metrics.
10
11 -The Netdata Agent uses a time-series database (TSDB), named the [database engine
12 -(`dbengine`)](/database/engine/README.md), to store metrics data. The most recently-collected metrics are stored in RAM,
13 -and when metrics reach a certain age, and based on how much system RAM you allocate toward storing metrics in memory,
14 -they are compressed and "spilled" to disk for long-term storage.
11 +The default settings retain approximately two day's worth of metrics on a system collecting 2,000 metrics every second,
12 +but the Netdata Agent is highly configurable if you want your nodes to store days, weeks, or months worth of per-second
13 +data.
14
16 -The default settings retain about two day's worth of metrics on a system collecting 2,000 metrics every second, but the
17 -Netdata Agent is highly configurable if you want your nodes to store days, weeks, or months worth of per-second data.
18 -
19 -The Netdata Agent uses two settings in `netdata.conf` to change the behavior of the database engine:
15 +The Netdata Agent uses the following three fundamental settings in `netdata.conf` to change the behavior of the database engine:
16
17 ```conf
18 [global]
23 - page cache size = 32
19 + dbengine page cache size = 32
20 dbengine multihost disk space = 256
21 + storage tiers = 1
22 ```
23
27 -`page cache size` sets the maximum amount of RAM (in MiB) the database engine uses to cache and index recent metrics.
24 +`dbengine page cache size` sets the maximum amount of RAM (in MiB) the database engine uses to cache and index recent
25 +metrics.
26 `dbengine multihost disk space` sets the maximum disk space (again, in MiB) the database engine uses to store
29 -historical, compressed metrics. When the size of stored metrics exceeds the allocated disk space, the database engine
30 -removes the oldest metrics on a rolling basis.
27 +historical, compressed metrics and `storage tiers` specifies the number of storage tiers you want to have in
28 +your `dbengine`. When the size of stored metrics exceeds the allocated disk space, the database engine removes the
29 +oldest metrics on a rolling basis.
30
31 ## Calculate the system resources (RAM, disk space) needed to store metrics
32
33 You can store more or less metrics using the database engine by changing the allocated disk space. Use the calculator
35 -below to find an appropriate value for `dbengine multihost disk space` based on how many metrics your node(s) collect,
36 -whether you are streaming metrics to a parent node, and more.
34 +below to find the appropriate value for the `dbengine` based on how many metrics your node(s) collect, whether you are
35 +streaming metrics to a parent node, and more.
36 +
37 +You do not need to edit the `dbengine page cache size` setting to store more metrics using the database engine. However,
38 +if you want to store more metrics _specifically in memory_, you can increase the cache size.
39 +
40 +:::tip
41 +
42 +We advise you to visit the [tiering mechanism](/database/engine/README.md#tiering) reference. This will help you
43 +configure the Agent to retain metrics for longer periods.
44
38 -You do not need to edit the `page cache size` setting to store more metrics using the database engine. However, if you
39 -want to store more metrics _specifically in memory_, you can increase the cache size.
45 +:::
46
41 -> ⚠️ This calculator provides an estimate of disk and RAM usage for **metrics storage**, along with its best
42 -> recommendation for the `dbengine multihost disk space` setting. Real-life usage may vary based on the accuracy of the
43 -> values you enter below, changes in the compression ratio, and the types of metrics stored.
47 +:::caution
48
45 -<Calculator />
49 +This calculator provides an estimation of disk and RAM usage for **metrics usage**. Real-life usage may vary based on
50 +the accuracy of the values you enter below, changes in the compression ratio, and the types of metrics stored.
51 +
52 +:::
53 +
54 +Download
55 +the [calculator](https://docs.google.com/spreadsheets/d/e/2PACX-1vTYMhUU90aOnIQ7qF6iIk6tXps57wmY9lxS6qDXznNJrzCKMDzxU3zkgh8Uv0xj_XqwFl3U6aHDZ6ag/pub?output=xlsx)
56 +to optimize the data retention to your preferences. Utilize the "Front" spreadsheet. Experiment with the variables which
57 +are padded with yellow to come up with the best settings for your use case.
58
59 ## Edit `netdata.conf` with recommended database engine settings
60
49 -Now that you have a recommended setting for `dbengine multihost disk space`, open `netdata.conf` with
50 -[`edit-config`](/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) and look for the `dbengine
51 -multihost disk space` setting. Change it to the value recommended above. For example:
61 +Now that you have a recommended setting for your Agent's `dbengine`, open `netdata.conf` with
62 +[`edit-config`](/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) and look for the `[db]`
63 +subsection. Change it to the recommended values you calculated from the calculator. For example:
64
65 ```conf
54 -[global]
55 - dbengine multihost disk space = 1024
66 +[db]
67 + mode = dbengine
68 + storage tiers = 3
69 + update every = 1
70 + dbengine multihost disk space MB = 1024
71 + dbengine page cache size MB = 32
72 + dbengine tier 1 update every iterations = 60
73 + dbengine tier 1 multihost disk space MB = 384
74 + dbengine tier 1 page cache size MB = 32
75 + dbengine tier 2 update every iterations = 60
76 + dbengine tier 2 multihost disk space MB = 16
77 + dbengine tier 2 page cache size MB = 32
78 ```
79
58 -Save the file and restart the Agent with `sudo systemctl restart netdata`, or the [appropriate
59 -method](/docs/configure/start-stop-restart.md) for your system, to change the database engine's size.
80 +Save the file and restart the Agent with `sudo systemctl restart netdata`, or
81 +the [appropriate method](/docs/configure/start-stop-restart.md) for your system, to change the database engine's size.
82
83 ## What's next?
84
63 -If you have multiple nodes with the Netdata Agent installed, you can [stream
64 -metrics](/docs/metrics-storage-management/how-streaming-works.mdx) from any number of _child_ nodes to a _parent_ node
65 -and store metrics using a centralized time-series database. Streaming allows you to centralize your data, run Agents as
66 -headless collectors, replicate data, and more.
85 +If you have multiple nodes with the Netdata Agent installed, you
86 +can [stream metrics](/docs/metrics-storage-management/how-streaming-works.mdx) from any number of _child_ nodes to a _
87 +parent_ node and store metrics using a centralized time-series database. Streaming allows you to centralize your data,
88 +run Agents as headless collectors, replicate data, and more.
89
68 -Storing metrics with the database engine is completely interoperable with [exporting to other time-series
69 -databases](/docs/export/external-databases.md). With exporting, you can use the node's resources to surface metrics
70 -when [viewing dashboards](/docs/visualize/interact-dashboards-charts.md), while also archiving metrics elsewhere for
71 -further analysis, visualization, or correlation with other tools.
90 +Storing metrics with the database engine is completely interoperable
91 +with [exporting to other time-series databases](/docs/export/external-databases.md). With exporting, you can use the
92 +node's resources to surface metrics when [viewing dashboards](/docs/visualize/interact-dashboards-charts.md), while also
93 +archiving metrics elsewhere for further analysis, visualization, or correlation with other tools.
94
95 ### Related reference documentation
96
97 - [Netdata Agent · Database engine](/database/engine/README.md)
98 +- [Netdata Agent · Database engine configuration option](/daemon/config/README.md#[db]-section-options)
99
100
docs/store/distributed-data-architecture.md
+42 -25
@@ -10,34 +10,43 @@ Netdata uses a distributed data architecture to help you collect and store per-s
10 Every node in your infrastructure, whether it's one or a thousand, stores the metrics it collects.
11
12 Netdata Cloud bridges the gap between many distributed databases by _centralizing the interface_ you use to query and
13 -visualize your nodes' metrics. When you [look at charts in Netdata
14 -Cloud](/docs/visualize/interact-dashboards-charts.md), the metrics values are queried directly from that node's database
15 -and securely streamed to Netdata Cloud, which proxies them to your browser.
13 +visualize your nodes' metrics. When you [look at charts in Netdata Cloud](/docs/visualize/interact-dashboards-charts.md)
14 +, the metrics values are queried directly from that node's database and securely streamed to Netdata Cloud, which
15 +proxies them to your browser.
16
17 Netdata's distributed data architecture has a number of benefits:
18
19 -- **Performance**: Every query to a node's database takes only a few milliseconds to complete for responsiveness when
20 - viewing dashboards or using features like [Metric
21 - Correlations](https://learn.netdata.cloud/docs/cloud/insights/metric-correlations).
22 -- **Scalability**: As your infrastructure scales, install the Netdata Agent on every new node to immediately add it to
23 - your monitoring solution without adding cost or complexity.
24 -- **1-second granularity**: Without an expensive centralized data lake, you can store all of your nodes' per-second
25 - metrics, for any period of time, while keeping costs down.
26 -- **No filtering or selecting of metrics**: Because Netdata's distributed data architecture allows you to store all
27 - metrics, you don't have to configure which metrics you retain. Keep everything for full visibility during
28 - troubleshooting and root cause analysis.
29 -- **Easy maintenance**: There is no centralized data lake to purchase, allocate, monitor, and update, removing
30 - complexity from your monitoring infrastructure.
19 +- **Performance**: Every query to a node's database takes only a few milliseconds to complete for responsiveness when
20 + viewing dashboards or using features
21 + like [Metric Correlations](https://learn.netdata.cloud/docs/cloud/insights/metric-correlations).
22 +- **Scalability**: As your infrastructure scales, install the Netdata Agent on every new node to immediately add it to
23 + your monitoring solution without adding cost or complexity.
24 +- **1-second granularity**: Without an expensive centralized data lake, you can store all of your nodes' per-second
25 + metrics, for any period of time, while keeping costs down.
26 +- **No filtering or selecting of metrics**: Because Netdata's distributed data architecture allows you to store all
27 + metrics, you don't have to configure which metrics you retain. Keep everything for full visibility during
28 + troubleshooting and root cause analysis.
29 +- **Easy maintenance**: There is no centralized data lake to purchase, allocate, monitor, and update, removing
30 + complexity from your monitoring infrastructure.
31
32 -## Does Netdata Cloud store my metrics?
32 +## Ephemerality of metrics
33
34 -Netdata Cloud does not store metric values.
34 +The ephemerality of metrics plays an important role in retention. In environments where metrics collection is dynamic and
35 +new metrics are constantly being generated, we are interested about 2 parameters:
36
36 -To enable certain features, such as [viewing active alarms](/docs/monitor/view-active-alarms.md) or [filtering by
37 -hostname/service](https://learn.netdata.cloud/docs/cloud/war-rooms#node-filter), Netdata Cloud does store configured
38 -alarms, their status, and a list of active collectors.
37 +1. The **expected concurrent number of metrics** as an average for the lifetime of the database. This affects mainly the
38 + storage requirements.
39
40 -Netdata does not and never will sell your personal data or data about your deployment.
40 +2. The **expected total number of unique metrics** for the lifetime of the database. This affects mainly the memory
41 + requirements for having all these metrics indexed and available to be queried.
42 +
43 +## Granularity of metrics
44 +
45 +The granularity of metrics (the frequency they are collected and stored, i.e. their resolution) is significantly
46 +affecting retention.
47 +
48 +Lowering the granularity from per second to every two seconds, will double their retention and half the CPU requirements
49 +of the Netdata Agent, without affecting disk space or memory requirements.
50
51 ## Long-term metrics storage with Netdata
52
@@ -47,7 +56,8 @@ appropriate amount of RAM and disk space.
56 Read our document on changing [how long Netdata stores metrics](/docs/store/change-metrics-storage.md) on your nodes for
57 details.
58
50 -## Other options for your metrics data
59 +You can also stream between nodes using [streaming](/streaming/README.md), allowing to replicate databases and create
60 +your own centralized data lake of metrics, if you choose to do so.
61
62 While a distributed data architecture is the default when monitoring infrastructure with Netdata, you can also configure
63 its behavior based on your needs or the type of infrastructure you manage.
@@ -55,12 +65,19 @@ its behavior based on your needs or the type of infrastructure you manage.
65 To archive metrics to an external time-series database, such as InfluxDB, Graphite, OpenTSDB, Elasticsearch,
66 TimescaleDB, and many others, see details on [integrating Netdata via exporting](/docs/export/external-databases.md).
67
58 -You can also stream between nodes using [streaming](/streaming/README.md), allowing to replicate databases and create
59 -your own centralized data lake of metrics, if you choose to do so.
60 -
68 When you use the database engine to store your metrics, you can always perform a quick backup of a node's
69 `/var/cache/netdata/dbengine/` folder using the tool of your choice.
70
71 +## Does Netdata Cloud store my metrics?
72 +
73 +Netdata Cloud does not store metric values.
74 +
75 +To enable certain features, such as [viewing active alarms](/docs/monitor/view-active-alarms.md)
76 +or [filtering by hostname/service](https://learn.netdata.cloud/docs/cloud/war-rooms#node-filter), Netdata Cloud does
77 +store configured alarms, their status, and a list of active collectors.
78 +
79 +Netdata does not and never will sell your personal data or data about your deployment.
80 +
81 ## What's next?
82
83 You can configure the Netdata Agent to store days, weeks, or months worth of distributed, per-second data by