Update documentation about dbengine (#6264)
* Update README.md with more up-to-date information regarding dbengine
Markos Fountoulakis committed
Jun 21, 2019 at 13:31 UTC
965ae842f3bfb093349b0dfe26e34a1f05622ebb
3 files changed
+64
-13
database/README.md
+7
-3
@@ -87,7 +87,9 @@ server that will maintain the entire database for all nodes, and will also run h
87
for all nodes.
88
89
For this central netdata, memory size can be a problem. Fortunately, netdata supports several
90
-memory modes. One interesting option for this setup is `memory mode = map`.
90
+memory modes. **One interesting option** for this setup is `memory mode = map`.
91
+
92
+### map
93
94
In this mode, the database of netdata is stored in memory mapped files. netdata continues to read
95
and write the database in memory, but the kernel automatically loads and saves memory pages from/to
@@ -162,8 +164,10 @@ vm.dirty_ratio = 90
164
vm.dirty_writeback_centisecs = 0
165
```
166
165
-There is another memory mode to help overcome the memory size problem. What is most interesting
166
-for this setup is `memory mode = dbengine`.
167
+There is another memory mode to help overcome the memory size problem. What is **most interesting
168
+for this setup** is `memory mode = dbengine`.
169
+
170
+### dbengine
171
172
In this mode, the database of netdata is stored in database files. The [Database Engine](engine/)
173
works like a traditional database. There is some amount of RAM dedicated to data caching and
database/engine/README.md
+37
@@ -106,4 +106,41 @@ There are explicit memory requirements **per** DB engine **instance**, meaning *
106
An important observation is that RAM usage depends on both the `page cache size` and the
107
`dbengine disk space` options.
108
109
+## File descriptor requirements
110
+
111
+The Database Engine may keep a **significant** amount of files open per instance (e.g. per streaming
112
+slave or master server). When configuring your system you should make sure there are at least 50
113
+file descriptors available per `dbengine` instance.
114
+
115
+Netdata allocates 25% of the available file descriptors to its Database Engine instances. This means that only 25%
116
+of the file descriptors that are available to the Netdata service are accessible by dbengine instances.
117
+You should take that into account when configuring your service
118
+or system-wide file descriptor limits. You can roughly estimate that the netdata service needs 2048 file
119
+descriptors for every 10 streaming slave hosts when streaming is configured to use `memory mode = dbengine`.
120
+
121
+If for example one wants to allocate 65536 file descriptors to the netdata service on a systemd system
122
+one needs to edit the netdata systemd file `netdata.service` in `/lib/systemd/system` or `/usr/lib/systemd/system`
123
+and add the line:
124
+
125
+```
126
+LimitNOFILE=65536
127
+```
128
+
129
+in the `[Service]` section.
130
+
131
+For other types of services one can add the line:
132
+```
133
+ulimit -n 65536
134
+```
135
+at the beginning of the service file. Alternatively you can change the system-wide limits of the kernel by changing `/etc/sysctl.conf`. For linux that would be:
136
+```
137
+fs.file-max = 65536
138
+```
139
+In FreeBSD and OS X you change the lines like this:
140
+```
141
+kern.maxfilesperproc=65536
142
+kern.maxfiles=65536
143
+```
144
+You can apply the settings by running `sysctl -p` or by rebooting.
145
+
146
[]()
packaging/installer/README.md
+20
-10
@@ -165,13 +165,25 @@ To install the latest git version of Netdata, please follow these 2 steps:
165
166
Try our experimental automatic requirements installer (no need to be root). This will try to find the packages that should be installed on your system to build and run Netdata. It supports most major Linux distributions released after 2010:
167
168
-- **Alpine** Linux and its derivatives (you have to install `bash` yourself, before using the installer)
169
-- **Arch** Linux and its derivatives
170
-- **Gentoo** Linux and its derivatives
171
-- **Debian** Linux and its derivatives (including **Ubuntu**, **Mint**)
172
-- **Redhat Enterprise Linux** and its derivatives (including **Fedora**, **CentOS**, **Amazon Machine Image**)
173
-- **SuSe** Linux and its derivatives (including **openSuSe**)
174
-- **SLE12** Must have your system registered with Suse Customer Center or have the DVD. See [#1162](https://github.com/netdata/netdata/issues/1162)
168
+* **Alpine** Linux and its derivatives
169
+ * You have to install `bash` yourself, before using the installer.
170
+
171
+* **Arch** Linux and its derivatives
172
+ * You need arch/aur for package Judy.
173
+
174
+* **Gentoo** Linux and its derivatives
175
+
176
+* **Debian** Linux and its derivatives (including **Ubuntu**, **Mint**)
177
+
178
+* **Redhat Enterprise Linux** and its derivatives (including **Fedora**, **CentOS**, **Amazon Machine Image**)
179
+ * Please note that for RHEL/CentOS you need
180
+ [EPEL](http://www.tecmint.com/how-to-enable-epel-repository-for-rhel-centos-6-5/).
181
+ In addition, RHEL/CentOS version 6 also need
182
+ [OKay](https://okay.com.mx/blog-news/rpm-repositories-for-centos-6-and-7.html) for package libuv version 1.
183
+
184
+* **SuSe** Linux and its derivatives (including **openSuSe**)
185
+
186
+* **SLE12** Must have your system registered with Suse Customer Center or have the DVD. See [#1162](https://github.com/netdata/netdata/issues/1162)
187
188
Install the packages for having a **basic Netdata installation** (system monitoring and many applications, without `mysql` / `mariadb`, `postgres`, `named`, hardware sensors and `SNMP`):
189
@@ -202,12 +214,10 @@ dnf install zlib-devel libuuid-devel libuv-devel lz4-devel Judy-devel openssl-de
214
yum install autoconf automake curl gcc git libmnl-devel libuuid-devel openssl-devel libuv-devel lz4-devel Judy-devel lm_sensors make MySQL-python nc pkgconfig python python-psycopg2 PyYAML zlib-devel
215
216
# openSUSE
205
-zypper install zlib-devel libuuid-devel libuv-devel liblz4-devel judy-devel openssl-devel libmnl-devel gcc make git autoconf autoconf-archive autogen automake pkgconfig curl findutils
217
+zypper install zlib-devel libuuid-devel libuv-devel liblz4-devel judy-devel libopenssl-devel libmnl-devel gcc make git autoconf autoconf-archive autogen automake pkgconfig curl findutils
218
219
```
220
209
-Please note that for RHEL/CentOS you might need [EPEL](http://www.tecmint.com/how-to-enable-epel-repository-for-rhel-centos-6-5/).
210
-
221
Once Netdata is compiled, to run it the following packages are required (already installed using the above commands):
222
223
package|description