Add documentation to provide a comprehensive guide for package maintainers (#9467)
* Initial commit of manual build guide. * Clean up header per suggestion from @joelhans Co-authored-by: Joel Hans <joel.g.hans@gmail.com> * Add GA tag per request from @joelhans Co-authored-by: Joel Hans <joel.g.hans@gmail.com> * Add simple description to metadata tags. * Elaborate on why we need a custom libmosquitto. * Update wording WRT manual install. * Add more info about LWS requirements. * Formatting fixes. * Fix typo. Co-authored-by: Joel Hans <joel.g.hans@gmail.com>
Austin S. Hemmelgarn committed
Jul 30, 2020 at 13:23 UTC
74824cc88f04b7043c9b0587f41dc6929d075afd
3 files changed
+300
-5
packaging/installer/README.md
+7
-3
@@ -95,9 +95,9 @@ Netdata on macOS](/packaging/installer/methods/macos.md)
95
FreeBSD](https://user-images.githubusercontent.com/1153921/76029787-5fc40580-5ef2-11ea-9461-23e9049aa8f8.png) Install
96
Netdata on FreeBSD](/packaging/installer/methods/freebsd.md)
97
98
-[ Install
100
-manually from source](/packaging/installer/methods/manual.md)
98
+[ Install
100
+from a Git checkout](/packaging/installer/methods/manual.md)
101
102
[ Install on
@@ -119,6 +119,10 @@ installation on FreeNAS](/packaging/installer/methods/freenas.md)
119
Alpine](https://user-images.githubusercontent.com/1153921/76029682-37d4a200-5ef2-11ea-9a2c-a8ffeb1d13c3.png) Manual
120
installation on Alpine](/packaging/installer/methods/alpine.md)
121
122
+[
124
+Build manually from source](/packaging/installer/methods/source.md)
125
+
126
</div>
127
128
## Automatic updates
packaging/installer/methods/manual.md
+2
-2
@@ -1,11 +1,11 @@
1
<!--
2
---
3
-title: "Install Netdata on Linux manually"
3
+title: "Install Netdata on Linux from a Git checkout"
4
custom_edit_url: https://github.com/netdata/netdata/edit/master/packaging/installer/methods/manual.md
5
---
6
-->
7
8
-# Install Netdata on Linux manually
8
+# Install Netdata on Linux from a Git checkout
9
10
To install the latest git version of Netdata, please follow these 2 steps:
11
packaging/installer/methods/source.md
new
+291
@@ -0,0 +1,291 @@
1
+<!--
2
+title: "Manually build Netdata from source"
3
+custom_edit_url: https://github.com/netdata/netdata/edit/master/packaging/installer/methods/source.md
4
+description: "Package maintainers and power users may be interested in manually building Netdata from source without using any of our installation scripts."
5
+-->
6
+
7
+# Manually build Netdata from source
8
+
9
+These instructions are for advanced users and distribution package
10
+maintainers. Unless this describes you, you almost certainly want
11
+to follow [our guide for manually installing Netdata from a git
12
+checkout](/packaging/installer/methods/manual.md) instead.
13
+
14
+## Required dependencies
15
+
16
+At a bare minimum, Netdata requires the following libraries and tools
17
+to build and run successfully:
18
+
19
+- libuuid
20
+- libuv version 1.0 or newer
21
+- zlib
22
+- GNU autoconf
23
+- GNU automake
24
+- GCC or Xcode (Clang is known to have issues in certain configurations, see [Using Clang](#using-clang))
25
+- A version of `make` compatible with GNU automake
26
+- Git (we use git in the build system to generate version info, don't need a full install, just a working `git show` command)
27
+
28
+Additionally, the following build time features require additional dependencies:
29
+
30
+- TLS support for the web GUI:
31
+ - OpenSSL 1.0.2 or newer _or_ LibreSSL 3.0.0 or newer.
32
+- dbengine metric storage:
33
+ - liblz4 r129 or newer
34
+ - OpenSSL 1.0 or newer (LibreSSL _amy_ work, but is largely untested).
35
+ - [libJudy](http://judy.sourceforge.net/)
36
+- Netdata Cloud support:
37
+ - A working internet connection
38
+ - A recent version of CMake
39
+ - OpenSSL 1.0.2 or newer _or_ LibreSSL 3.0.0 or newer.
40
+ - JSON-C (may be provided by the user as shown below, or by the system)
41
+
42
+## Preparing the source tree
43
+
44
+Certain features in Netdata require custom versions of specific libraries,
45
+which the the build system will link statically into Netdata. These
46
+libraries and their header files must be copied into specific locations
47
+in the source tree to be used.
48
+
49
+### Netdata cloud
50
+
51
+Netdata Cloud functionality requires custom builds of libmosquitto and
52
+libwebsockets.
53
+
54
+#### libmosquitto
55
+
56
+Netdata maintains a custom fork of libmosquitto at
57
+https://github.com/netdata/mosquitto with patches to allow for proper
58
+integration with libwebsockets, which is needed for correct operation of
59
+Netdata Cloud functionality. To prepare this library for the build system:
60
+
61
+1. Verify the tag that Netdata expects to be used by checking the contents
62
+ of `packaging/mosquitto.version` in your Netdata sources.
63
+2. Obtain the sources for that version by either:
64
+ - Navigating to https://github.com/netdata/mosquitto/releases and
65
+ donwloading and unpacking the source code archive for that release.
66
+ - Cloning the repository with `git` and checking out the required tag.
67
+3. If building on a platfom other than Linux, prepare the mosquitto
68
+ sources by running `cmake -D WITH_STATIC_LIBRARIES:boolean=YES .` in
69
+ the mosquitto source directory.
70
+4. Build mosquitto by running `make -C lib` in the mosquitto source directory.
71
+5. In the Netdata source directory, create a directory called `externaldeps/mosquitto`.
72
+6. Copy `lib/mosquitto.h` from the mosquitto source directory to
73
+ `externaldeps/mosquitto/mosquitto.h` in the Netdata source tree.
74
+7. Copy `lib/libmosquitto.a` from the mosquitto source directory to
75
+ `externaldeps/mosquitto/libmosquitto.a` in the Netdata source tree. If
76
+ building on a platform other than Linux, the file that needs to be
77
+ copied will instead be named `lib/libmosquitto_static.a`, but it
78
+ still needs to be copied to `externaldeps/mosquitto/libmosquitto.a`.
79
+
80
+#### libwebsockets
81
+
82
+Netdata uses the standard upstream version of libwebsockets located at
83
+https://github.com/warmcat/libwebsockets, but requires a build with SOCKS5
84
+support, which is not enabled by most pre-built versions. Currently,
85
+we do not support using a system copy of libwebsockets. To prepare this
86
+library for the build system:
87
+
88
+1. Verify the tag that Netdata expects to be used by checking the contents
89
+ of `packaging/libwebsockets.version` in your Netdata sources.
90
+2. Obtain the sources for that version by either:
91
+ - Navigating to https://github.com/warmcat/libwebsockets/releases and
92
+ donwloading and unpacking the source code archive for that release.
93
+ - Cloning the repository with `git` and checking out the required tag.
94
+3. Prepare the libweboskcets sources by running `cmake -D
95
+ LWS_WITH_SOCKS5:bool=ON .` in the libwebsockets source directory.
96
+4. Build libwebsockets by running `make` in the libwebsockets source
97
+ directory.
98
+5. In the Netdata source directory, create a directory called
99
+ `externaldeps/libwebsockets`.
100
+6. Copy `lib/libwebsockets.a` from the libwebsockets source directory to
101
+ `externaldeps/libwebsockets/libwebsockets.a` in the Netdata source tree.
102
+7. Copy the entire contents of `lib/include` from the libwebsockets source
103
+ directory to `externaldeps/libwebsockets/include` in the Netdata source tree.
104
+
105
+#### JSON-C
106
+
107
+Netdata requires the use of JSON-C for JSON parsing when using Netdata
108
+Cloud. Netdata is able to use a system-provided copy of JSON-C, but
109
+some systems may not provide it. If your system does not provide JSON-C,
110
+you can do the following to prepare a copy for the build system:
111
+
112
+1. Verify the tag that Netdata expects to be used by checking the contents
113
+ of `packaging/jsonc.version` in your Netdata sources.
114
+2. Obtain the sources for that version by either:
115
+ - Navigating to https://github.com/json-c/json-c and donwloading
116
+ and unpacking the source code archive for that release.
117
+ - Cloning the repository with `git` and checking out the required tag.
118
+3. Prepare the JSON-C sources by running `cmake -DBUILD_SHARED_LIBS=OFF .`
119
+ in the JSON-C source directory.
120
+4. Build JSON-C by running `make` in the JSON-C source directory.
121
+5. In the Netdata source directory, create a directory called
122
+ `externaldeps/jsonc`.
123
+6. Copy `libjson-c.a` fro the JSON-C source directory to
124
+ `externaldeps/jsonc/libjson-c.a` in the Netdata source tree.
125
+7. Copy all of the header files (`*.h`) from the JSON-C source directory
126
+ to `externaldeps/jsonc/json-c` in the Netdata source tree.
127
+
128
+## Building Netdata
129
+
130
+Once the source tree has been prepared, Netdata is ready to be configured
131
+and built. Netdata currently uses GNU autotools as it's primary build
132
+system. To build Netdata this way:
133
+
134
+1. Run `autoreconf -ivf` in the Netdata source tree.
135
+2. Run `./configure` in the Netdata source tree.
136
+3. Run `make` in the Netdata source tree.
137
+
138
+### Configure options
139
+
140
+Netdata provides a number of build time configure options. This section
141
+lists some of the ones you are most likely to need:
142
+
143
+- `--prefix`: Specify the prefix under which Netdata will be installed.
144
+- `--with-webdir`: Specify a path relative to the prefix in which to
145
+ install the web UI files.
146
+- `--disable-cloud`: Disables all Netdata Cloud functionality for
147
+ this build.
148
+
149
+### Using Clang
150
+
151
+Netdata is primarily developed using GCC, but in most cases we also
152
+build just fine using Clang. Under some build configurations of Clang
153
+itself, you may see build failures with the linker reporting errors
154
+about `nonrepresentable section on output`. We currently do not have a
155
+conclusive fix for this issue (the obvious fix leads to other issues which
156
+we haven't been able to fix yet), and unfortunately the only workaround
157
+is to use a different build of Clang or to use GCC.
158
+
159
+### Linking errors relating to OpenSSL
160
+
161
+Netdata's build system currently does not reliably support building
162
+on systems which have multiple ABI incompatible versions of OpenSSL
163
+installed. In such situations, you may encounter linking errors due to
164
+Netdata trying to build against headers for one version but link to a
165
+different version.
166
+
167
+## Additional components
168
+
169
+A full featured install of Netdata requires some additional components
170
+which must be built and installed separately from the main Netdata
171
+agent. All of these should be handled _after_ installing Netdata itself.
172
+
173
+### React dashboard
174
+
175
+The above build steps include a deprecated web UI for Netdata that lacks
176
+support for Netdata Cloud. To get a fully featured dashboard, you must
177
+install our new React dashboard.
178
+
179
+#### Installing the pre-built React dashboard
180
+
181
+We provide pre-built archives of the React dashboard for each release
182
+(these are also used during our normal install process). To use one
183
+of these:
184
+
185
+1. Verify the release version that Netdata expects to be used by checking
186
+ the contents of `packaging/dashboard.version` in your Netdata sources.
187
+2. Go to https://github.com/netdata/dashboard/releases and download the
188
+ `dashboard.tar.gz` file for the required release.
189
+3. Unpack the downloaded archive to a temporary directory.
190
+4. Copy the contents of the `build` directory from the extracted
191
+ archive to `/usr/share/netdata/web` or the equivalent location for
192
+ your build of Netdata. This _will_ overwrite some files in the target
193
+ location.
194
+
195
+#### Building the React dashboard locally
196
+
197
+Alternatively, you may wish to build the React dashboard locally. Doing
198
+so requires a recent version of Node.JS with a working install of
199
+NPM. Once you have the required tools, do the following:
200
+
201
+1. Verify the release version that Netdata expects to be used by checking
202
+ the contents of `packaging/dashboard.version` in your Netdata sources.
203
+2. Obtain the sources for that version by either:
204
+ - Navigating to https://github.com/netdata/dashboard and donwloading
205
+ and unpacking the source code archive for that release.
206
+ - Cloning the repository with `git` and checking out the required tag.
207
+3. Run `npm install` in the dashboard source tree.
208
+4. Run `npm run build` in the dashboard source tree.
209
+5. Copy the contents of the `build` directory just like step 4 of
210
+ installing the pre-built React dashboard.
211
+
212
+### Go collectors
213
+
214
+A number of the collectors for Netdata are written in Go instead of C,
215
+and are developed in a separate repository from the mian Netdata code.
216
+An installation without these collectors is still usable, but will be
217
+unable to collect metrics for a number of network services the system
218
+may be providing. You can either install a pre-built copy of these
219
+eollectors, or build them locally.
220
+
221
+#### Installing the pre-built Go collectors
222
+
223
+We provide pre-built binaries of the Go collectors for all the platforms
224
+we officially support. To use one of these:
225
+
226
+1. Verify the release version that Netdata expects to be used by checking
227
+ the contents of `packaging/go.d.version` in your Netdata sources.
228
+2. Go to https://github.com/netdata/go.d.plugin/releases, select the
229
+ required release, and download the `go.d.plugin-*.tar.gz` file
230
+ for your system type and CPu architecture and the `config.tar.gz`
231
+ configuration file archive.
232
+3. Extract the `go.d.plugin-*.tar.gz` archive into a temprary
233
+ location, and then copy the single file in the archive to
234
+ `/usr/libexec/netdata/plugins.d` or the equivalent location for your
235
+ build of Netdata and rename it to `go.d.plugin`.
236
+4. Extract the `config.tar.gz` archive to a temporarylocation and then
237
+ copy the contents of the archive to `/etc/netdata` or the equivalent
238
+ location for your build of Netdata.
239
+
240
+#### Building the Go collectors locally
241
+
242
+Alternatively, you may wish to build the Go collectors locally
243
+yourself. Doing so requires a working installation of Golang 1.13 or
244
+newer. Once you have the required tools, do the following:
245
+
246
+1. Verify the release version that Netdata expects to be used by checking
247
+ the contents of `packaging/go.d.version` in your Netdata sources.
248
+2. Obtain the sources for that version by either:
249
+ - Navigating to https://github.com/netdata/go.d.plugin and donwloading
250
+ and unpacking the source code archive for that release.
251
+ - Cloning the repository with `git` and checking out the required tag.
252
+3. Run `make` in the go.d.plugin source tree.
253
+4. Copy `bin/godplugin` to `/usr/libexec/netdata/plugins.d` or th
254
+ eequivalent location for your build of Netdata and rename it to
255
+ `go.d.plugin`.
256
+5. Copy the contents of the `config` directory to `/etc/netdata` or the
257
+ equivalent location for your build of Netdata.
258
+
259
+### eBPF collector
260
+
261
+On Linux systems, Netdata has support for using the kernel's eBPF
262
+interface to monitor performance-related VFS, network, and process events,
263
+allowing for insights into process lifetimes and file access
264
+patterns. Using this functionality requires additional code managed in
265
+a separate repository from the core Netdata agent. You can either install
266
+a pre-built copy of the required code, or build it locally.
267
+
268
+#### Installing the pre-built eBPF code
269
+
270
+We provide pre-built copies of the eBPF code for 64-bit x86 systems
271
+using glibc or musl. To use one of these:
272
+
273
+1. Verify the release version that Netdata expects to be used by checking
274
+ the contents of `packaging/ebpf.version` in your Netdata sources.
275
+2. Go to https://github.com/netdata/kernel-collector/releases, select the
276
+ required release, and download the `netdata-kernel-collector-*.tar.xz`
277
+ file for the libc variant your system uses (eithe rmusl or glibc).
278
+3. Extract the contents of the archive to a temporary location, and then
279
+ copy all of the `.o` and `.so.*` files and the contents of the `library/`
280
+ directory to `/usr/libexec/netdata/plugins.d` or the equivalent location
281
+ for your build of Netdata.
282
+
283
+#### Building the eBPF code locally
284
+
285
+Alternatively, you may wish to build the eBPF code locally yourself. For
286
+instructions, please consult [the README file for our kernel-collector
287
+repository](https://github.com/netdata/kernel-collector/blob/master/README.md),
288
+which outlines both the required dependencies, as well as multiple
289
+options for building the code.
290
+
291
+[](<>)