netdata/installer: Add support for offline installs using kickstart or kickstart-static64 (#6693)
* netdata/installer: Introduce parameter, for offiline installs 1) introduce parameter --local-tarball-override that takes two files as input 2) make kickstart and kickstart-static64 handle this parameter by overriding the download process and use the given tarballs * netdata/installer: update checksums * netdata/installer: attempt to update readme.md * netdata/installer: [ci skip] add documentation for static64 * netdata/installer: Offline installation -- expect go.d plugin binaries also for the intall to proceed 1) Introduce two more artifacts on the tarball override parameter (go.d plugin tarball and go.d plugin config tarball) 2) Adjust the installer to expect the offline install, when the path is set use that file instead of downloading anything * Update packaging/installer/README.md Co-Authored-By: Joel Hans <joel.g.hans@gmail.com> * Update packaging/installer/README.md Co-Authored-By: Joel Hans <joel.g.hans@gmail.com> * netdata/installer: add offline installer section * netdata/installer: don't shift there, we shift by one when needed * netdata/installer: Update md5sum * netdata/installer: adjust proposed commands to fetch proper binary based on OS and ARCH * netdata/packaging: first round of feedback * netdata/packaging: dont remove the files provided by the users * netdata/packaging: local-tarball-override -> local-files 1) rename parameter name 2) update md5sums in readme * netdata/packaging: fixes and nits 1) checksumi -> checksum 2) we are using sh, so conditionals are with single equal sign (fixed it) 3) unfortunately add one more option to the --local-files parameter, to define the required packages script. Might make sense in the end to just pass a directory structure there and inform the user of the files we expect to see there * netdata/packaging: update md5sum * netdata/packaging: do not move, copy * netdata/packaging: heres a few more missed mv->cp fixes * netdata/packaging: [ci skip] Refine error message on kickstart-static64.sh, to help cases when user confuses kickstart.sh with kickstart-static64.sh * netdata/packaging: update documentation to properly instruct on offline install how t orun the kickstart itself * netdata/packaging: fix documentation - missing one more script call
Paul Emm. Katsoulakis committed
Sep 17, 2019 at 18:01 UTC
4dc34c078d5ae030d444e820c1f50ae533ac540d
4 files changed
+196
-22
netdata-installer.sh
+12
-2
@@ -807,9 +807,19 @@ install_go() {
807
tmp=$(mktemp -d /tmp/netdata-go-XXXXXX)
808
GO_PACKAGE_BASENAME="go.d.plugin-${GO_PACKAGE_VERSION}.${OS}-${ARCH}.tar.gz"
809
810
- download_go "https://github.com/netdata/go.d.plugin/releases/download/${GO_PACKAGE_VERSION}/${GO_PACKAGE_BASENAME}" "${tmp}/${GO_PACKAGE_BASENAME}"
810
+ if [ -z "${NETDATA_LOCAL_TARBALL_OVERRIDE_GO_PLUGIN}" ]; then
811
+ download_go "https://github.com/netdata/go.d.plugin/releases/download/${GO_PACKAGE_VERSION}/${GO_PACKAGE_BASENAME}" "${tmp}/${GO_PACKAGE_BASENAME}"
812
+ else
813
+ progress "Using provided go.d tarball ${NETDATA_LOCAL_TARBALL_OVERRIDE_GO_PLUGIN}"
814
+ run cp "${NETDATA_LOCAL_TARBALL_OVERRIDE_GO_PLUGIN}" "${tmp}/${GO_PACKAGE_BASENAME}"
815
+ fi
816
812
- download_go "https://github.com/netdata/go.d.plugin/releases/download/${GO_PACKAGE_VERSION}/config.tar.gz" "${tmp}/config.tar.gz"
817
+ if [ -z "${NETDATA_LOCAL_TARBALL_OVERRIDE_GO_PLUGIN_CONFIG}" ]; then
818
+ download_go "https://github.com/netdata/go.d.plugin/releases/download/${GO_PACKAGE_VERSION}/config.tar.gz" "${tmp}/config.tar.gz"
819
+ else
820
+ progress "Using provided config file for go.d ${NETDATA_LOCAL_TARBALL_OVERRIDE_GO_PLUGIN_CONFIG}"
821
+ run cp "${NETDATA_LOCAL_TARBALL_OVERRIDE_GO_PLUGIN_CONFIG}" "${tmp}/config.tar.gz"
822
+ fi
823
824
if [ ! -f "${tmp}/${GO_PACKAGE_BASENAME}" ] || [ ! -f "${tmp}/config.tar.gz" ] || [ ! -s "${tmp}/config.tar.gz" ] || [ ! -s "${tmp}/${GO_PACKAGE_BASENAME}" ]; then
825
run_failed "go.d plugin download failed, go.d plugin will not be available"
packaging/installer/README.md
+78
-5
@@ -48,7 +48,7 @@ To learn more about the pros and cons of using *nightly* vs. *stable* releases,
48
Verify the integrity of the script with this:
49
50
```bash
51
-[ "b6d16c171ccad073b86327246151d875" = "$(curl -Ss https://my-netdata.io/kickstart.sh | md5sum | cut -d ' ' -f 1)" ] && echo "OK, VALID" || echo "FAILED, INVALID"
51
+[ "735e9966a4cf0187863e06a5282b34a7" = "$(curl -Ss https://my-netdata.io/kickstart.sh | md5sum | cut -d ' ' -f 1)" ] && echo "OK, VALID" || echo "FAILED, INVALID"
52
```
53
54
_It should print `OK, VALID` if the script is the one we ship._
@@ -67,13 +67,14 @@ The `kickstart.sh` script passes all its parameters to `netdata-installer.sh`, s
67
- `--dont-start-it`: Prevent the installer from starting Netdata automatically.
68
- `--stable-channel`: Automatically update only on the release of new major versions.
69
- `--no-updates`: Prevent automatic updates of any kind.
70
+- `--local-files`: Used for offline installations. Pass four file paths: the Netdata tarball, the checksum file, the go.d plugin tarball, and the go.d plugin config tarball, to force kickstart run the process using those files.
71
72
Example using all the above parameters:
73
74
```bash
74
-$ bash <(curl -Ss https://my-netdata.io/kickstart.sh) --dont-wait --dont-start-it --no-updates --stable-channel
75
+$ bash <(curl -Ss https://my-netdata.io/kickstart.sh) --dont-wait --dont-start-it --no-updates --stable-channel --local-files /tmp/my-selfdownloaded-tarball.tar.gz /tmp/checksums.txt /tmp/manually.downloaded.go.d.binary.tar.gz /tmp/manually.downloaded.go.d.config.tar.gz
76
```
76
-
77
+Note: `--stable-channel` and `--local-files` overlap, if you use the tarball override the stable channel option is not effective
78
</details>
79
80
Once Netdata is installed, see [Getting Started](../../docs/GettingStarted.md).
@@ -108,7 +109,7 @@ This script installs Netdata at `/opt/netdata`.
109
Verify the integrity of the script with this:
110
111
```bash
111
-[ "4415e8c13e529a795abb953a9be14ad5" = "$(curl -Ss https://my-netdata.io/kickstart-static64.sh | md5sum | cut -d ' ' -f 1)" ] && echo "OK, VALID" || echo "FAILED, INVALID"
112
+[ "5c432e78d5f40403e8b7c00b1a19a7ca" = "$(curl -Ss https://my-netdata.io/kickstart-static64.sh | md5sum | cut -d ' ' -f 1)" ] && echo "OK, VALID" || echo "FAILED, INVALID"
113
```
114
115
*It should print `OK, VALID` if the script is the one we ship.*
@@ -119,12 +120,14 @@ The `kickstart-static64.sh` script passes all its parameters to `netdata-install
120
- `--dont-start-it`: Prevent the installer from starting Netdata automatically.
121
- `--stable-channel`: Automatically update only on the release of new major versions.
122
- `--no-updates`: Prevent automatic updates of any kind.
123
+- `--local-files`: Used for offline installations. Pass two file paths, one for the tarball and one fir the checksum file, to force kickstart run the process using those files.
124
125
Example using all the above parameters:
126
127
```sh
126
-$ bash <(curl -Ss https://my-netdata.io/kickstart-static64.sh) --dont-wait --dont-start-it --no-updates --stable-channel
128
+$ bash <(curl -Ss https://my-netdata.io/kickstart-static64.sh) --dont-wait --dont-start-it --no-updates --stable-channel --local-files /tmp/my-selfdownloaded-tarball.tar.gz /tmp/checksums.txt
129
```
130
+Note: `--stable-channel` and `--local-files` overlap, if you use the tarball override the stable channel option is not effective
131
132
If your shell fails to handle the above one liner, do this:
133
@@ -504,6 +507,76 @@ The Netdata team maintains two releases of the Netdata agent: **nightly** and **
507
- Protect yourself from the rare instance when major bugs slip through our testing and negatively affect a Netdata installation
508
- Retain more control over the Netdata version you use
509
510
+## Offline installations
511
+
512
+You can install Netdata on systems without internet access, but you need to take
513
+a few extra steps to make it work.
514
+
515
+By default, the `kickstart.sh` and `kickstart-static64.sh` download Netdata
516
+assets, like the precompiled binary and a few dependencies, using the system's
517
+internet connection, but you can also supply these files from the local filesystem.
518
+
519
+First, download the required files. If you're using `kickstart.sh`, you need the
520
+Netdata tarball, the checksums, the go.d plugin binary, and the go.d plugin
521
+configuration. If you're using `kickstart-static64.sh`, you need only the
522
+Netdata tarball and checksums.
523
+
524
+Download the files you need to a system of yours that's connected to the
525
+internet. You can use the commands below, or visit the [latest Netdata release
526
+page](https://github.com/netdata/netdata/releases/latest) and [latest go.d
527
+plugin release page](https://github.com/netdata/go.d.plugin/releases) to
528
+download the required files manually.
529
+
530
+```bash
531
+cd /tmp
532
+
533
+# Since you won't be having an active internet connection on the destination, you will need to manually get the kickstart script itself
534
+# Download kickstart.sh or kickstart-static64.sh, depending the installation you have chosen to run
535
+curl -s https://my-netdata.io/kickstart.sh > kickstart.sh
536
+or
537
+curl -s https://my-netdata.io/kickstart-static64.sh > kickstart-static64.sh
538
+
539
+# Netdata tarball
540
+curl -s https://api.github.com/repos/netdata/netdata/releases/latest | grep "browser_download_url.*tar.gz" | cut -d '"' -f 4 | wget -qi -
541
+
542
+# Netdata checksums
543
+curl -s https://api.github.com/repos/netdata/netdata/releases/latest | grep "browser_download_url.*txt" | cut -d '"' -f 4 | wget -qi -
544
+
545
+# Netdata dependency handling script
546
+curl -s https://raw.githubusercontent.com/netdata/netdata-demo-site/master/install-required-packages.sh | wget -qi -
547
+
548
+# go.d plugin
549
+# For binaries for OS types and architectures not listed on [go.d releases](https://github.com/netdata/go.d.plugin/releases/latest), kindly open a github issue and we will do our best to serve your request
550
+export OS=$(uname -s | tr '[:upper:]' '[:lower:]') ARCH=$(uname -m | sed -e 's/i386/386/g' -e 's/i686/386/g' -e 's/x86_64/amd64/g' -e 's/aarch64/arm64/g' -e 's/armv64/arm64/g' -e 's/armv6l/arm/g' -e 's/armv7l/arm/g' -e 's/armv5tel/arm/g') && curl -s https://api.github.com/repos/netdata/go.d.plugin/releases/latest | grep "browser_download_url.*${OS}-${ARCH}.tar.gz" | cut -d '"' -f 4 | wget -qi -
551
+
552
+# go.d configuration
553
+curl -s https://api.github.com/repos/netdata/go.d.plugin/releases/latest | grep "browser_download_url.*config.tar.gz" | cut -d '"' -f 4 | wget -qi -
554
+```
555
+
556
+Move these files to the `/tmp` directory on the offline system in whichever way
557
+your defined policy allows (if any).
558
+
559
+Now you can run either the `kickstart.sh` or `kickstart-static64.sh` scripts
560
+using the `--local-files` option. This option requires you to specify
561
+the location and names of the files you just downloaded.
562
+
563
+!!! note When using `--local-files`, the `kickstart.sh` or
564
+ `kickstart-static64.sh` scripts won't download any Netdata assets from the
565
+ internet. But, you may still need a connection to install dependencies using
566
+ your system's package manager. The scripts will warn you if your system
567
+ doesn't have all the dependencies.
568
+
569
+```bash
570
+# kickstart.sh
571
+bash kickstart.sh --local-files /tmp/netdata-version-number-here.tar.gz /tmp/sha256sums.txt /tmp/go.d-binary-filename.tar.gz /tmp/config.tar.gz /tmp/install-required-packages.sh
572
+
573
+# kickstart-static64.sh
574
+bash kickstart-static64.sh --local-files /tmp/netdata.tar.gz /tmp/checksums.txt
575
+```
576
+
577
+Now that you're finished with your offline installation, you can move on to our
578
+[getting started guide](../../docs/GettingStarted.md)!
579
+
580
## Automatic updates
581
582
By default, Netdata's installation scripts enable automatic updates for both nightly and stable release channels.
packaging/installer/kickstart-static64.sh
+40
-7
@@ -1,7 +1,15 @@
1
#!/usr/bin/env sh
2
+#
3
# SPDX-License-Identifier: GPL-3.0-or-later
4
# shellcheck disable=SC1117,SC2039,SC2059,SC2086
4
-
5
+#
6
+# Options to run
7
+# --dont-wait do not wait for input
8
+# --non-interactive do not wait for input
9
+# --dont-start-it do not start netdata after install
10
+# --stable-channel Use the stable release channel, rather than the nightly to fetch sources
11
+# --local-files Use a manually provided tarball for the installation
12
+#
13
# ---------------------------------------------------------------------------------------------------------------------
14
# library functions copied from packaging/installer/functions.sh
15
@@ -127,6 +135,11 @@ download() {
135
}
136
137
set_tarball_urls() {
138
+ if [ -n "${NETDATA_LOCAL_TARBALL_OVERRIDE}" ]; then
139
+ progress "Not fetching remote tarballs, local override was given"
140
+ return
141
+ fi
142
+
143
if [ "$1" = "stable" ]; then
144
local latest
145
# Simple version
@@ -179,11 +192,24 @@ while [ -n "${1}" ]; do
192
inner_opts="${inner_opts} ${1}"
193
elif [ "${1}" = "--stable-channel" ]; then
194
RELEASE_CHANNEL="stable"
195
+ elif [ "${1}" = "--local-files" ]; then
196
+ shift 1
197
+ if [ -z "${1}" ]; then
198
+ fatal "Option --local-files requires extra information. The desired tarball full filename is needed"
199
+ fi
200
+
201
+ NETDATA_LOCAL_TARBALL_OVERRIDE="${1}"
202
+ shift 1
203
+ if [ -z "${1}" ]; then
204
+ fatal "Option --local-files requires a pair of the tarball source and the checksum file"
205
+ fi
206
+
207
+ NETDATA_LOCAL_TARBALL_OVERRIDE_CHECKSUM="${1}"
208
+ shift 1
209
else
183
- echo >&2 "Unknown option '${1}'"
210
+ echo >&2 "Unknown option '${1}' or invalid number of arguments. Please check the README for the available arguments of ${0} and try again"
211
exit 1
212
fi
186
- shift
213
done
214
[ -n "${inner_opts}" ] && inner_opts="-- ${inner_opts}"
215
@@ -191,11 +217,18 @@ done
217
TMPDIR=$(create_tmp_directory)
218
cd "${TMPDIR}"
219
194
-set_tarball_urls "${RELEASE_CHANNEL}"
195
-progress "Downloading static netdata binary: ${NETDATA_TARBALL_URL}"
220
+if [ -z "${NETDATA_LOCAL_TARBALL_OVERRIDE}" ]; then
221
+ set_tarball_urls "${RELEASE_CHANNEL}"
222
+ progress "Downloading static netdata binary: ${NETDATA_TARBALL_URL}"
223
+
224
+ download "${NETDATA_TARBALL_CHECKSUM_URL}" "${TMPDIR}/sha256sum.txt"
225
+ download "${NETDATA_TARBALL_URL}" "${TMPDIR}/netdata-latest.gz.run"
226
+else
227
+ progress "Installation sources were given as input, running installation using \"${NETDATA_LOCAL_TARBALL_OVERRIDE}\""
228
+ run cp "${NETDATA_LOCAL_TARBALL_OVERRIDE}" "${TMPDIR}/netdata-latest.gz.run"
229
+ run cp "${NETDATA_LOCAL_TARBALL_OVERRIDE_CHECKSUM}" "${TMPDIR}/sha256sum.txt"
230
+fi
231
197
-download "${NETDATA_TARBALL_CHECKSUM_URL}" "${TMPDIR}/sha256sum.txt"
198
-download "${NETDATA_TARBALL_URL}" "${TMPDIR}/netdata-latest.gz.run"
232
if ! grep netdata-latest.gz.run "${TMPDIR}/sha256sum.txt" | safe_sha256sum -c - >/dev/null 2>&1; then
233
fatal "Static binary checksum validation failed. Stopping netdata installation and leaving binary in ${TMPDIR}"
234
fi
packaging/installer/kickstart.sh
+66
-8
@@ -10,9 +10,10 @@
10
# bash <(curl -Ss https://my-netdata.io/kickstart.sh) all
11
#
12
# Other options:
13
-# --dont-wait do not prompt for user input
14
-# --non-interactive do not prompt for user input
15
-# --no-updates do not install script for daily updates
13
+# --dont-wait do not prompt for user input
14
+# --non-interactive do not prompt for user input
15
+# --no-updates do not install script for daily updates
16
+# --local-files set the full path of the desired tarball to run install with
17
#
18
# This script will:
19
#
@@ -163,6 +164,11 @@ download() {
164
}
165
166
set_tarball_urls() {
167
+ if [ -n "${NETDATA_LOCAL_TARBALL_OVERRIDE}" ]; then
168
+ progress "Not fetching remote tarballs, local override was given"
169
+ return
170
+ fi
171
+
172
if [ "$1" = "stable" ]; then
173
local latest
174
# Simple version
@@ -216,8 +222,17 @@ dependencies() {
222
if ! detect_bash4 "${bash}"; then
223
warning "Cannot detect packages to be installed in this system, without BASH v4+."
224
else
219
- progress "Downloading script to detect required packages..."
220
- download "${PACKAGES_SCRIPT}" "${TMPDIR}/install-required-packages.sh"
225
+ progress "Fetching script to detect required packages..."
226
+ if [ -n "${NETDATA_LOCAL_TARBALL_OVERRIDE_DEPS_SCRIPT}" ]; then
227
+ if [ -f "${NETDATA_LOCAL_TARBALL_OVERRIDE_DEPS_SCRIPT}" ]; then
228
+ run cp "${NETDATA_LOCAL_TARBALL_OVERRIDE_DEPS_SCRIPT}" "${TMPDIR}/install-required-packages.sh"
229
+ else
230
+ fatal "Invalid given dependency file, please check your --local-files parameter options and try again"
231
+ fi
232
+ else
233
+ download "${PACKAGES_SCRIPT}" "${TMPDIR}/install-required-packages.sh"
234
+ fi
235
+
236
if [ ! -s "${TMPDIR}/install-required-packages.sh" ]; then
237
warning "Downloaded dependency installation script is empty."
238
else
@@ -293,6 +308,42 @@ while [ -n "${1}" ]; do
308
elif [ "${1}" = "--stable-channel" ]; then
309
RELEASE_CHANNEL="stable"
310
shift 1
311
+ elif [ "${1}" = "--local-files" ]; then
312
+ shift 1
313
+ if [ -z "${1}" ]; then
314
+ fatal "Missing netdata: Option --local-files requires extra information. The desired tarball for netdata, the checksum, the go.d plugin tarball , the go.d plugin config tarball and the dependency management script, in this particular order"
315
+ fi
316
+
317
+ export NETDATA_LOCAL_TARBALL_OVERRIDE="${1}"
318
+ shift 1
319
+
320
+ if [ -z "${1}" ]; then
321
+ fatal "Missing checksum file: Option --local-files requires extra information. The desired tarball for netdata, the checksum, the go.d plugin tarball , the go.d plugin config tarball and the dependency management script, in this particular order"
322
+ fi
323
+
324
+ export NETDATA_LOCAL_TARBALL_OVERRIDE_CHECKSUM="${1}"
325
+ shift 1
326
+
327
+ if [ -z "${1}" ]; then
328
+ fatal "Missing go.d tarball: Option --local-files requires extra information. The desired tarball for netdata, the checksum, the go.d plugin tarball , the go.d plugin config tarball and the dependency management script, in this particular order"
329
+ fi
330
+
331
+ export NETDATA_LOCAL_TARBALL_OVERRIDE_GO_PLUGIN="${1}"
332
+ shift 1
333
+
334
+ if [ -z "${1}" ]; then
335
+ fatal "Missing go.d config tarball: Option --local-files requires extra information. The desired tarball for netdata, the checksum, the go.d plugin tarball , the go.d plugin config tarball and the dependency management script, in this particular order"
336
+ fi
337
+
338
+ export NETDATA_LOCAL_TARBALL_OVERRIDE_GO_PLUGIN_CONFIG="${1}"
339
+ shift 1
340
+
341
+ if [ -z "${1}" ]; then
342
+ fatal "Missing dependencies management scriptlet: Option --local-files requires extra information. The desired tarball for netdata, the checksum, the go.d plugin tarball , the go.d plugin config tarball and the dependency management script, in this particular order"
343
+ fi
344
+
345
+ export NETDATA_LOCAL_TARBALL_OVERRIDE_DEPS_SCRIPT="${1}"
346
+ shift 1
347
else
348
break
349
fi
@@ -311,10 +362,17 @@ dependencies
362
# ---------------------------------------------------------------------------------------------------------------------
363
# download netdata package
364
314
-set_tarball_urls "${RELEASE_CHANNEL}"
365
+if [ -z "${NETDATA_LOCAL_TARBALL_OVERRIDE}" ]; then
366
+ set_tarball_urls "${RELEASE_CHANNEL}"
367
+
368
+ download "${NETDATA_TARBALL_CHECKSUM_URL}" "${TMPDIR}/sha256sum.txt"
369
+ download "${NETDATA_TARBALL_URL}" "${TMPDIR}/netdata-latest.tar.gz"
370
+else
371
+ progress "Installation sources were given as input, running installation using \"${NETDATA_LOCAL_TARBALL_OVERRIDE}\""
372
+ run cp "${NETDATA_LOCAL_TARBALL_OVERRIDE_CHECKSUM}" "${TMPDIR}/sha256sum.txt"
373
+ run cp "${NETDATA_LOCAL_TARBALL_OVERRIDE}" "${TMPDIR}/netdata-latest.tar.gz"
374
+fi
375
316
-download "${NETDATA_TARBALL_CHECKSUM_URL}" "${TMPDIR}/sha256sum.txt"
317
-download "${NETDATA_TARBALL_URL}" "${TMPDIR}/netdata-latest.tar.gz"
376
if ! grep netdata-latest.tar.gz "${TMPDIR}/sha256sum.txt" | safe_sha256sum -c - >/dev/null 2>&1; then
377
fatal "Tarball checksum validation failed. Stopping netdata installation and leaving tarball in ${TMPDIR}"
378
fi