885caf08d · cryptotaxi247/netdata-1

@@ -6,52 +6,28 @@ custom_edit_url: https://github.com/netdata/netdata/edit/master/database/engine/

6

7 # Database engine

8

9 -The Database Engine works like a traditional database. There is some amount of RAM dedicated to data caching and

10 -indexing and the rest of the data reside compressed on disk. The number of history entries is not fixed in this case,

11 -but depends on the configured disk space and the effective compression ratio of the data stored. This is the **only

12 -mode** that supports changing the data collection update frequency (`update_every`) **without losing** the previously

13 -stored metrics.

9 +The Database Engine works like a traditional database. It dedicates a certain amount of RAM to data caching and

10 +indexing, while the rest of the data resides compressed on disk. Unlike other [memory modes](/database/README.md), the

11 +amount of historical metrics stored is based on the amount of disk space you allocate and the effective compression

12 +ratio, not a fixed number of metrics collected.

13

15 -## Files

14 +By using both RAM and disk space, the database engine allows for long-term storage of per-second metrics inside of the

15 +Agent itself.

16

17 -With the DB engine memory mode the metric data are stored in database files. These files are organized in pairs, the

18 -datafiles and their corresponding journalfiles, e.g.:

19 -

20 -```sh

21 -datafile-1-0000000001.ndf

22 -journalfile-1-0000000001.njf

23 -datafile-1-0000000002.ndf

24 -journalfile-1-0000000002.njf

25 -datafile-1-0000000003.ndf

26 -journalfile-1-0000000003.njf

27 -...

28 -```

29 -

30 -They are located under their host's cache directory in the directory `./dbengine` (e.g. for localhost the default

31 -location is `/var/cache/netdata/dbengine/*`). The higher numbered filenames contain more recent metric data. The user

32 -can safely delete some pairs of files when Netdata is stopped to manually free up some space.

33 -

34 -_Users should_ **back up** _their `./dbengine` folders if they consider this data to be important._

17 +In addition, the database engine is the only memory mode that supports changing the data collection update frequency

18 +(`update_every`) without losing the metrics your Agent already gathered and stored.

19

20 ## Configuration

21

38 -There is one DB engine instance per Netdata host/node. That is, there is one `./dbengine` folder per node, and all

39 -charts of `dbengine` memory mode in such a host share the same storage space and DB engine instance memory state. You

40 -can select the memory mode for localhost by editing netdata.conf and setting:

22 +To use the database engine, open `netdata.conf` and set `memory mode` to `dbengine`.

23

24 ```conf

25 [global]

26 memory mode = dbengine

27 ```

28

47 -For setting the memory mode for the rest of the nodes you should look at

48 -[streaming](/streaming/README.md).

49 -

50 -The `history` configuration option is meaningless for `memory mode = dbengine` and is ignored for any metrics being

51 -stored in the DB engine.

52 -

53 -All DB engine instances, for localhost and all other streaming recipient nodes inherit their configuration from

54 -`netdata.conf`:

29 +To configure the database engine, look for the `page cache size` and `dbengine disk space` settings in the `[global]`

30 +section of your `netdata.conf`. The Agent ignores the `history` setting when using the database engine.

31

32 ```conf

33 [global]

@@ -60,38 +36,36 @@ All DB engine instances, for localhost and all other streaming recipient nodes i

36 ```

37

38 The above values are the default and minimum values for Page Cache size and DB engine disk space quota. Both numbers are

63 -in **MiB**. All DB engine instances will allocate the configured resources separately.

64 -

65 -[**See our database engine calculator**](https://learn.netdata.cloud/docs/agent/database/calculator) to help you

66 -correctly set `dbengine disk space` based on your needs. The calculator gives an accurate estimate based on how many

67 -slave nodes you have, how many metrics your Agent collects, and more.

39 +in **MiB**.

40

69 -The `page cache size` option determines the amount of RAM in **MiB** that is dedicated to caching Netdata metric values

70 -themselves as far as queries are concerned. The total page cache size will be greater since data collection itself will

71 -consume additional memory as is described in the [Memory requirements](#memory-requirements) section.

41 +The `page cache size` option determines the amount of RAM in **MiB** dedicated to caching Netdata metric values. The

42 +actual page cache size will be slightly larger than this figure—see the [memory requirements](#memory-requirements)

43 +section for details.

44

45 The `dbengine disk space` option determines the amount of disk space in **MiB** that is dedicated to storing Netdata

46 metric values and all related metadata describing them.

47

76 -## Operation

48 +Use the [**database engine calculator**](https://learn.netdata.cloud/docs/agent/database/calculator) to correctly set

49 +`dbengine disk space` based on your needs. The calculator gives an accurate estimate based on how many slave nodes you

50 +have, how many metrics your Agent collects, and more.

51

78 -The DB engine stores chart metric values in 4096-byte pages in memory. Each chart dimension gets its own page to store

79 -consecutive values generated from the data collectors. Those pages comprise the **Page Cache**.

52 +### Streaming metrics to the database engine

53

81 -When those pages fill up they are slowly compressed and flushed to disk. It can take `4096 / 4 = 1024 seconds = 17

82 -minutes`, for a chart dimension that is being collected every 1 second, to fill a page. Pages can be cut short when we

83 -stop Netdata or the DB engine instance so as to not lose the data. When we query the DB engine for data we trigger disk

84 -read I/O requests that fill the Page Cache with the requested pages and potentially evict cold (not recently used)

85 -pages.

54 +When streaming metrics, the Agent on the master node creates one instance of the database engine for itself, and another

55 +instance for every slave node it receives metrics from. If you have four streaming nodes, you will have five instances

56 +in total (`1 master + 4 slaves = 5 instances`).

57

87 -When the disk quota is exceeded the oldest values are removed from the DB engine at real time, by automatically deleting

88 -the oldest datafile and journalfile pair. Any corresponding pages residing in the Page Cache will also be invalidated

89 -and removed. The DB engine logic will try to maintain between 10 and 20 file pairs at any point in time.

58 +The Agent allocates resources for each instance separately using the `dbengine disk space` setting. If `dbengine disk

59 +space` is set to the default `256`, each instance is given 256 MiB in disk space, which means the total disk space

60 +required to store all instances is, roughly, `256 MiB * 1 master * 4 slaves = 1280 MiB`.

61

91 -The Database Engine uses direct I/O to avoid polluting the OS filesystem caches and does not generate excessive I/O

92 -traffic so as to create the minimum possible interference with other applications.

62 +See the [database engine calculator](https://learn.netdata.cloud/docs/agent/database/calculator) to help you correctly

63 +set `dbengine disk space` and undertand the toal disk space required based on your streaming setup.

64 +

65 +For more information about setting `memory mode` on your nodes, in addition to other streaming configurations, see

66 +[streaming](/streaming/README.md).

67

94 -## Memory requirements

68 +### Memory requirements

69

70 Using memory mode `dbengine` we can overcome most memory restrictions and store a dataset that is much larger than the

71 available memory.

@@ -114,7 +88,7 @@ An important observation is that RAM usage depends on both the `page cache size`

88 You can use our [database engine calculator](https://learn.netdata.cloud/docs/agent/database/calculator) to

89 validate the memory requirements for your particular system(s) and configuration.

90

117 -## File descriptor requirements

91 +### File descriptor requirements

92

93 The Database Engine may keep a **significant** amount of files open per instance (e.g. per streaming slave or master

94 server). When configuring your system you should make sure there are at least 50 file descriptors available per

@@ -156,6 +130,47 @@ kern.maxfiles=65536

130

131 You can apply the settings by running `sysctl -p` or by rebooting.

132

133 +## Files

134 +

135 +With the DB engine memory mode the metric data are stored in database files. These files are organized in pairs, the

136 +datafiles and their corresponding journalfiles, e.g.:

137 +

138 +```sh

139 +datafile-1-0000000001.ndf

140 +journalfile-1-0000000001.njf

141 +datafile-1-0000000002.ndf

142 +journalfile-1-0000000002.njf

143 +datafile-1-0000000003.ndf

144 +journalfile-1-0000000003.njf

145 +...

146 +```

147 +

148 +They are located under their host's cache directory in the directory `./dbengine` (e.g. for localhost the default

149 +location is `/var/cache/netdata/dbengine/*`). The higher numbered filenames contain more recent metric data. The user

150 +can safely delete some pairs of files when Netdata is stopped to manually free up some space.

151 +

152 +_Users should_ **back up** _their `./dbengine` folders if they consider this data to be important._ You can also set up

153 +one or more [exporting connectors](/exporting/README.md) to send your Netdata metrics to other databases for long-term

154 +storage at lower granularity.

155 +

156 +## Operation

157 +

158 +The DB engine stores chart metric values in 4096-byte pages in memory. Each chart dimension gets its own page to store

159 +consecutive values generated from the data collectors. Those pages comprise the **Page Cache**.

160 +

161 +When those pages fill up they are slowly compressed and flushed to disk. It can take `4096 / 4 = 1024 seconds = 17

162 +minutes`, for a chart dimension that is being collected every 1 second, to fill a page. Pages can be cut short when we

163 +stop Netdata or the DB engine instance so as to not lose the data. When we query the DB engine for data we trigger disk

164 +read I/O requests that fill the Page Cache with the requested pages and potentially evict cold (not recently used)

165 +pages.

166 +

167 +When the disk quota is exceeded the oldest values are removed from the DB engine at real time, by automatically deleting

168 +the oldest datafile and journalfile pair. Any corresponding pages residing in the Page Cache will also be invalidated

169 +and removed. The DB engine logic will try to maintain between 10 and 20 file pairs at any point in time.

170 +

171 +The Database Engine uses direct I/O to avoid polluting the OS filesystem caches and does not generate excessive I/O

172 +traffic so as to create the minimum possible interference with other applications.

173 +

174 ## Evaluation

175

176 We have evaluated the performance of the `dbengine` API that the netdata daemon uses internally. This is **not** the

Improve dbengine docs for streaming setups (#9177)