95fe304f0 · cryptotaxi247/netdata-1

@@ -1,13 +1,3 @@

1 -<!--

2 -title: "Database engine"

3 -description: "Netdata's highly-efficient database engine use both RAM and disk for distributed, long-term storage of per-second metrics."

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/database/engine/README.md"

5 -sidebar_label: "Database engine"

6 -learn_status: "Published"

7 -learn_topic_type: "Concepts"

8 -learn_rel_path: "Concepts"

9 --->

10 -

1 # Database engine

2

3 DBENGINE is the time-series database of Netdata.

@@ -124,7 +114,7 @@ data from lower tiers, so that the aggregation they provide will be accurate.

114 Configuring how the number of tiers and the disk space allocated to each tier is how you can

115 [change how long netdata stores metrics](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md).

116

127 -### Data Loss

117 +### Data loss

118

119 Until **hot pages** and **dirty pages** are **flushed** to disk they are at risk (e.g. due to a crash, or

120 power failure), as they are stored only in memory.

@@ -132,36 +122,9 @@ power failure), as they are stored only in memory.

122 The supported way of ensuring high data availability is the use of Netdata Parents to stream the data in real-time to

123 multiple other Netdata agents.

124

135 -## Memory Requirements

136 -

137 -DBENGINE memory is related to the number of metrics concurrently being collected, the retention of the metrics on disk in relation with the queries running, and the number of metrics for which retention is maintained.

138 -

139 -### Memory for concurrently collected metrics

125 +## Memory requirements and retention

126

141 -DBENGINE is automatically sized to use memory according to this equation:

142 -

143 -```

144 -memory in KiB = METRICS x (TIERS - 1) x 4KiB x 2 + 32768 KiB

145 -```

146 -

147 -Where:

148 -- `METRICS`: the maximum number of concurrently collected metrics (dimensions) from the time the agent started.

149 -- `TIERS`: the number of storage tiers configured, by default 3 ( `-1` when using 3+ tiers)

150 -- `x 2`, to accommodate room for flushing data to disk

151 -- `x 4KiB`, the data segment size of each metric

152 -- `+ 32768 KiB`, 32 MB for operational caches

153 -

154 -So, for 2000 metrics (dimensions) in 3 storage tiers:

155 -

156 -```

157 -memory for 2k metrics = 2000 x (3 - 1) x 4 KiB x 2 + 32768 KiB = 64 MiB

158 -```

159 -

160 -For 100k concurrently collected metrics in 3 storage tiers:

161 -

162 -```

163 -memory for 100k metrics = 100000 x (3 - 1) x 4 KiB x 2 + 32768 KiB = 1.6 GiB

164 -```

127 +See (change how long netdata stores metrics)[https://github.com/netdata/netdata/edit/master/docs/store/change-metrics-storage.md]

128

129 #### Exceptions

130

@@ -222,216 +185,6 @@ The time-ranges of the queries running control the amount of shared memory requi

185

186 DBENGINE uses 150 bytes of memory for every metric for which retention is maintained but is not currently being collected.

187

225 ----

226 -

227 ---- OLD DOCS BELOW THIS POINT ---

228 -

229 ----

230 -

231 -

232 -## Legacy configuration

233 -

234 -### v1.35.1 and prior

235 -

236 -These versions of the Agent do not support [Tiers](#Tiers). You could change the metric retention for the parent and

237 -all of its children only with the `dbengine multihost disk space MB` setting. This setting accounts the space allocation

238 -for the parent node and all of its children.

239 -

240 -To configure the database engine, look for the `page cache size MB` and `dbengine multihost disk space MB` settings in

241 -the `[db]` section of your `netdata.conf`.

242 -

243 -```conf

244 -[db]

245 - dbengine page cache size MB = 32

246 - dbengine multihost disk space MB = 256

247 -```

248 -

249 -### v1.23.2 and prior

250 -

251 -_For Netdata Agents earlier than v1.23.2_, the Agent on the parent node uses one dbengine instance for itself, and another instance for every child node it receives metrics from. If you had four streaming nodes, you would have five instances in total (`1 parent + 4 child nodes = 5 instances`).

252 -

253 -The Agent allocates resources for each instance separately using the `dbengine disk space MB` (**deprecated**) setting. If `dbengine disk space MB`(**deprecated**) is set to the default `256`, each instance is given 256 MiB in disk space, which means the total disk space required to store all instances is, roughly, `256 MiB * 1 parent * 4 child nodes = 1280 MiB`.

254 -

255 -#### Backward compatibility

256 -

257 -All existing metrics belonging to child nodes are automatically converted to legacy dbengine instances and the localhost

258 -metrics are transferred to the multihost dbengine instance.

259 -

260 -All new child nodes are automatically transferred to the multihost dbengine instance and share its page cache and disk

261 -space. If you want to migrate a child node from its legacy dbengine instance to the multihost dbengine instance, you

262 -must delete the instance's directory, which is located in `/var/cache/netdata/MACHINE_GUID/dbengine`, after stopping the

263 -Agent.

264 -

265 -##### Information

266 -

267 -For more information about setting `[db].mode` on your nodes, in addition to other streaming configurations, see

268 -[streaming](https://github.com/netdata/netdata/blob/master/streaming/README.md).

269 -

270 -## Requirements & limitations

271 -

272 -### Memory

273 -

274 -Using database mode `dbengine` we can overcome most memory restrictions and store a dataset that is much larger than the

275 -available memory.

276 -

277 -There are explicit memory requirements **per** DB engine **instance**:

278 -

279 -- The total page cache memory footprint will be an additional `#dimensions-being-collected x 4096 x 2` bytes over what

280 - the user configured with `dbengine page cache size MB`.

281 -

282 -

283 -- an additional `#pages-on-disk x 4096 x 0.03` bytes of RAM are allocated for metadata.

284 -

285 - - roughly speaking this is 3% of the uncompressed disk space taken by the DB files.

286 -

287 - - for very highly compressible data (compression ratio > 90%) this RAM overhead is comparable to the disk space

288 - footprint.

289 -

290 -An important observation is that RAM usage depends on both the `page cache size` and the `dbengine multihost disk space`

291 -options.

292 -

293 -You can use

294 -our [database engine calculator](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md#calculate-the-system-resources-ram-disk-space-needed-to-store-metrics)

295 -to validate the memory requirements for your particular system(s) and configuration (**out-of-date**).

296 -

297 -### Disk space

298 -

299 -There are explicit disk space requirements **per** DB engine **instance**:

300 -

301 -- The total disk space footprint will be the maximum between `#dimensions-being-collected x 4096 x 2` bytes or what the

302 - user configured with `dbengine multihost disk space` or `dbengine disk space`.

303 -

304 -### File descriptor

305 -

306 -The Database Engine may keep a **significant** amount of files open per instance (e.g. per streaming child or parent

307 -server). When configuring your system you should make sure there are at least 50 file descriptors available per

308 -`dbengine` instance.

309 -

310 -Netdata allocates 25% of the available file descriptors to its Database Engine instances. This means that only 25% of

311 -the file descriptors that are available to the Netdata service are accessible by dbengine instances. You should take

312 -that into account when configuring your service or system-wide file descriptor limits. You can roughly estimate that the

313 -Netdata service needs 2048 file descriptors for every 10 streaming child hosts when streaming is configured to use

314 -`[db].mode = dbengine`.

315 -

316 -If for example one wants to allocate 65536 file descriptors to the Netdata service on a systemd system one needs to

317 -override the Netdata service by running `sudo systemctl edit netdata` and creating a file with contents:

318 -

319 -```sh

320 -[Service]

321 -LimitNOFILE=65536

322 -```

323 -

324 -For other types of services one can add the line:

325 -

326 -```sh

327 -ulimit -n 65536

328 -```

329 -

330 -at the beginning of the service file. Alternatively you can change the system-wide limits of the kernel by changing

331 -`/etc/sysctl.conf`. For linux that would be:

332 -

333 -```conf

334 -fs.file-max = 65536

335 -```

336 -

337 -In FreeBSD and OS X you change the lines like this:

338 -

339 -```conf

340 -kern.maxfilesperproc=65536

341 -kern.maxfiles=65536

342 -```

343 -

344 -You can apply the settings by running `sysctl -p` or by rebooting.

345 -

346 -## Files

347 -

348 -With the DB engine mode the metric data are stored in database files. These files are organized in pairs, the datafiles

349 -and their corresponding journalfiles, e.g.:

350 -

351 -```sh

352 -datafile-1-0000000001.ndf

353 -journalfile-1-0000000001.njf

354 -datafile-1-0000000002.ndf

355 -journalfile-1-0000000002.njf

356 -datafile-1-0000000003.ndf

357 -journalfile-1-0000000003.njf

358 -...

359 -```

360 -

361 -They are located under their host's cache directory in the directory `./dbengine` (e.g. for localhost the default

362 -location is `/var/cache/netdata/dbengine/*`). The higher numbered filenames contain more recent metric data. The user

363 -can safely delete some pairs of files when Netdata is stopped to manually free up some space.

364 -

365 -_Users should_ **back up** _their `./dbengine` folders if they consider this data to be important._ You can also set up

366 -one or more [exporting connectors](https://github.com/netdata/netdata/blob/master/exporting/README.md) to send your Netdata metrics to other databases for long-term

367 -storage at lower granularity.

368 -

369 -## Operation

370 -

371 -The DB engine stores chart metric values in 4096-byte pages in memory. Each chart dimension gets its own page to store

372 -consecutive values generated from the data collectors. Those pages comprise the **Page Cache**.

373 -

374 -When those pages fill up, they are slowly compressed and flushed to disk. It can

375 -take `4096 / 4 = 1024 seconds = 17 minutes`, for a chart dimension that is being collected every 1 second, to fill a

376 -page. Pages can be cut short when we stop Netdata or the DB engine instance so as to not lose the data. When we query

377 -the DB engine for data we trigger disk read I/O requests that fill the Page Cache with the requested pages and

378 -potentially evict cold (not recently used)

379 -pages.

380 -

381 -When the disk quota is exceeded the oldest values are removed from the DB engine at real time, by automatically deleting

382 -the oldest datafile and journalfile pair. Any corresponding pages residing in the Page Cache will also be invalidated

383 -and removed. The DB engine logic will try to maintain between 10 and 20 file pairs at any point in time.

384 -

385 -The Database Engine uses direct I/O to avoid polluting the OS filesystem caches and does not generate excessive I/O

386 -traffic so as to create the minimum possible interference with other applications.

387 -

388 -## Evaluation

389 -

390 -We have evaluated the performance of the `dbengine` API that the netdata daemon uses internally. This is **not** the web

391 -API of netdata. Our benchmarks ran on a **single** `dbengine` instance, multiple of which can be running in a Netdata

392 -parent node. We used a server with an AMD Ryzen Threadripper 2950X 16-Core Processor and 2 disk drives, a Seagate

393 -Constellation ES.3 2TB magnetic HDD and a SAMSUNG MZQLB960HAJR-00007 960GB NAND Flash SSD.

394 -

395 -For our workload, we defined 32 charts with 128 metrics each, giving us a total of 4096 metrics. We defined 1 worker

396 -thread per chart (32 threads) that generates new data points with a data generation interval of 1 second. The time axis

397 -of the time-series is emulated and accelerated so that the worker threads can generate as many data points as possible

398 -without delays.

399 -

400 -We also defined 32 worker threads that perform queries on random metrics with semi-random time ranges. The starting time

401 -of the query is randomly selected between the beginning of the time-series and the time of the latest data point. The

402 -ending time is randomly selected between 1 second and 1 hour after the starting time. The pseudo-random numbers are

403 -generated with a uniform distribution.

404 -

405 -The data are written to the database at the same time as they are read from it. This is a concurrent read/write mixed

406 -workload with a duration of 60 seconds. The faster `dbengine` runs, the bigger the dataset size becomes since more data

407 -points will be generated. We set a page cache size of 64MiB for the two disk-bound scenarios. This way, the dataset size

408 -of the metric data is much bigger than the RAM that is being used for caching so as to trigger I/O requests most of the

409 -time. In our final scenario, we set the page cache size to 16 GiB. That way, the dataset fits in the page cache so as to

410 -avoid all disk bottlenecks.

411 -

412 -The reported numbers are the following:

413 -

415 -|:------:|:----------:|--------:|----------:|-----------:|

416 -| HDD | 64 MiB | 4.1 GiB | 813K | 18.0M |

417 -| SSD | 64 MiB | 9.8 GiB | 1.7M | 43.0M |

418 -| N/A | 16 GiB | 6.8 GiB | 118.2M | 30.2M |

419 -

420 -where "reads/sec" is the number of metric data points being read from the database via its API per second and

421 -"writes/sec" is the number of metric data points being written to the database per second.

422 -

423 -Notice that the HDD numbers are pretty high and not much slower than the SSD numbers. This is thanks to the database

424 -engine design being optimized for rotating media. In the database engine disk I/O requests are:

188

426 -- asynchronous to mask the high I/O latency of HDDs.

427 -- mostly large to reduce the amount of HDD seeking time.

428 -- mostly sequential to reduce the amount of HDD seeking time.

429 -- compressed to reduce the amount of required throughput.

189

431 -As a result, the HDD is not thousands of times slower than the SSD, which is typical for other workloads.

190

433 -An interesting observation to make is that the CPU-bound run (16 GiB page cache) generates fewer data than the SSD run

434 -(6.8 GiB vs 9.8 GiB). The reason is that the 32 reader threads in the SSD scenario are more frequently blocked by I/O,

435 -and generate a read load of 1.7M/sec, whereas in the CPU-bound scenario the read load is 70 times higher at 118M/sec.

436 -Consequently, there is a significant degree of interference by the reader threads, that slow down the writer threads.

437 -This is also possible because the interference effects are greater than the SSD impact on data generation throughput.

Update database engine readme (#14836)