@cryptotaxi247 / netdata-1 / commits / 8f6b2bba3

Integrity testing: Check published kickstart files integrity (#5689)

* netdata/packaging/ci: Integrity testing Introduce a scriptlet that validates kickstart integrity in my-netdata.io against the ones in the repo. Run this through the pipeline on a scheduled manner. Will refine the conditionals later, once i verify the stage is built up as expected * netdata/packaging/ci: remove conditionals first * netdata/packaging/ci: Adjust the names to something more appropriate. Run it along with nightlies * netdata/packager/ci: Cleanup checksum validation for kickstart files (continued) 1) merge validate_kickstart_integrity.sh and tests/installer/checksums.sh 2) run checksums at the new point on the pipeline 3) Change that unstable gitignore check and use a more file-agnostic check that depends only on git that we already require 4) Do not run the kickstart validation on the online website everywhere, only on the nightly runs * netdata/packaging/ci: First rounf of PR feedback adjustments 1) changes wordings as recommended 2) pass file info in parameter and use it in the wordings 3) as shellcheck suggests, use -n instead of ! -z. Makes sense actually, ! -z is kind of reverse logic that confuses More adjustments on a follow up commit * netdata/packaging/ci: Enable slack integration for kickstart validation We want to be notified in a timely manner when the kickstart on the website is outdated. Added a wrapper for incoming webhooks from slack and instead of failing the build we notify slack Added a debug message in the end of the script to validate the process which i will remove on a follow up commit * netdata/packaging/ci: fixes fix sourcing path missed the /, this new keyboard is a pain :p

Paul Katsoulakis committed Mar 25, 2019 at 10:05 UTC 8f6b2bba3e0d82b27b26b7da1022e49fe9e09482
3 files changed +68 -11
.travis.yml
+8
@@ -11,6 +11,8 @@ stages:
11 if: branch = master AND type != pull_request AND type != cron
12 - name: nightlies
13 if: branch = master AND type = cron
14 +- name: Integrity testing
15 + if: branch = master AND type = cron
16
17 jobs:
18 include:
@@ -25,6 +27,7 @@ jobs:
27 script: shellcheck --format=gcc $(find . -name '*.sh.in' -not -iwholename '*.git*')
28 - name: check checksums for kickstart files
29 script: ./tests/installer/checksums.sh
30 + env: LOCAL_ONLY="true"
31 - name: coverity
32 install: sudo apt-get install -y zlib1g-dev uuid-dev libipmimonitoring-dev libmnl-dev libnetfilter-acct-dev
33 script: ./coverity-install.sh && ./coverity-scan.sh || echo "Coverity failed :("
@@ -81,6 +84,11 @@ jobs:
84 skip_cleanup: true
85 local_dir: "artifacts"
86 after_deploy: rm -f .travis/gcs-credentials.json
87 +
88 + - stage: Integrity testing
89 + name: Kickstart files integrity testing
90 + script: ./tests/installer/checksums.sh
91 +
92 notifications:
93 webhooks: https://app.fossa.io/hooks/travisci
94 slack:
tests/installer/checksums.sh
+43 -11
@@ -1,19 +1,51 @@
1 #!/bin/bash
2 -
2 +#
3 +# Mechanism to validate kickstart files integrity status
4 +#
5 +# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
6 +#
7 +# Author : Pawel Krupa (pawel@netdata.cloud)
8 +# Author : Pavlos Emm. Katsoulakis (paul@netdata.cloud)
9 set -e
10
5 -if [ ! -f .gitignore ]; then
6 - echo "Run as ./tests/installer/$(basename "$0") from top level directory of git repository"
7 - exit 1
11 +# If we are not in netdata git repo, at the top level directory, fail
12 +TOP_LEVEL=$(basename "$(git rev-parse --show-toplevel 2> /dev/null || echo "")")
13 +CWD="$(git rev-parse --show-cdup 2> /dev/null || echo "")"
14 +if [ -n "$CWD" ] || [ ! "${TOP_LEVEL}" == "netdata" ]; then
15 + echo "Run as .travis/$(basename "$0") from top level directory of netdata git repository"
16 + echo "Kickstart validation process aborted"
17 + exit 1
18 fi
19
20 +README_DOC="packaging/installer/README.md"
21 +source ./tests/installer/slack.sh
22 +
23 for file in kickstart.sh kickstart-static64.sh; do
11 - OLD_CHECKSUM=$(grep "$file" packaging/installer/README.md | grep md5sum | cut -d '"' -f2)
12 - NEW_CHECKSUM="$(md5sum "packaging/installer/$file" | cut -d' ' -f1)"
13 - if [ "$OLD_CHECKSUM" != "$NEW_CHECKSUM" ]; then
14 - echo "Invalid checksum for $file in docs."
15 - echo "checksum in docs: $OLD_CHECKSUM"
16 - echo "current checksum: $NEW_CHECKSUM"
17 - exit 1
24 + README_MD5=$(grep "$file" $README_DOC | grep md5sum | cut -d '"' -f2)
25 + KICKSTART_URL="https://my-netdata.io/$file"
26 + KICKSTART="packaging/installer/$file"
27 + KICKSTART_MD5="$(md5sum "${KICKSTART}" | cut -d' ' -f1)"
28 + CALCULATED_MD5="$(curl -Ss ${KICKSTART_URL} | md5sum | cut -d ' ' -f 1)"
29 +
30 + # Conditionally run the website validation
31 + if [ -z "${LOCAL_ONLY}" ]; then
32 + echo "Validating ${KICKSTART_URL} against local file ${KICKSTART} with MD5 ${KICKSTART_MD5}.."
33 + if [ "$KICKSTART_MD5" == "$CALCULATED_MD5" ]; then
34 + echo "${KICKSTART_URL} looks fine"
35 + else
36 + post_message "Attention @group , ${KICKSTART_URL} md5sum does not match local file, it needs to be updated"
37 + fi
38 fi
39 +
40 + echo "Validating documentation for $file"
41 + if [ "$KICKSTART_MD5" != "$README_MD5" ]; then
42 + echo "Invalid checksum for $file in $README_DOC."
43 + echo "checksum in docs: $README_MD5"
44 + echo "current checksum: $KICKSTART_MD5"
45 + exit 2
46 + else
47 + echo "$file MD5Sum is well documented"
48 + fi
49 +
50 done
51 +echo "No problems found, exiting succesfully!"
tests/installer/slack.sh new
+17
@@ -0,0 +1,17 @@
1 +#!/bin/bash
2 +#
3 +# Simple incoming webhook for slack integration.
4 +#
5 +# The script expects the following parameters to be defined by the upper layer:
6 +# SLACK_INCOMING_WEBHOOK_URL
7 +# SLACK_BOT_NAME
8 +# SLACK_CHANNEL
9 +#
10 +# Copyright:
11 +#
12 +# Author: Pavlos Emm. Katsoulakis <paul@netdata.cloud
13 +
14 +post_message() {
15 + MESSAGE="$1"
16 + curl -X POST --data-urlencode "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_BOT_NAME}\", \"text\": \"${MESSAGE}\", \"icon_emoji\": \":space_invader:\"}" ${SLACK_INCOMING_WEBHOOK_URL}
17 +}