@cryptotaxi247 / netdata-1 / commits / a8ba6234f

Pipeline redesign master (#5885)

* netdata/packaging/ci: [WIP] Redesign packaging release * netdata/packaging/ci: Add explicitly os definition, also remove sourcing of slack, its already done and rename before_install -> install * netdata/packaging/ci: fix stray stage name * netdata/packaging/ci: style * netdata/packaging/ci: fix syntax for yaml * netdata/packaging/ci: Try removing this, doesnt seem to make much sense to be there (for now) * Revert "netdata/packaging/ci: Try removing this, doesnt seem to make much sense to be there (for now)" This reverts commit ad5556a3ccf6c11ebc9026942af8013d4d0ea666. wow travis - this is weird * netdata/packaging/ci: a few more changes * netdata/packaging/ci: This is totally experimental, will adjust after i test it * netdata/packaging/ci: Reinstate original design - the problem i am trying to solve is more travis-related and fundamental on the way travis works * netdata/packaging/ci: i wonder.. * Revert "netdata/packaging/ci: i wonder.." This reverts commit a160df4fcac866375e8319bf200f47263b2bcaa3. * netdata/packaging/ci: Fix broken items * netdata/packaging/ci: docker images are built and published independently. As long as tagging has taken place earlier, during packaging * netdata/packaging/ci: Following hints from https://github.com/travis-ci/travis-ci/issues/4681 and the mayhem triggered from this change So we adding the compiler, to see if it will get travis to build the right matrix. then linter broke on addons for some weird reason, so remove addons->apt->update too * netdata/packaging/ci: Now do the rest of the changes, since with addition of compiler, things got worse * netdata/packaging/ci: Ok, as expected the workaround from them did not work. Lets get creative though and assume they exclude what they added, so lets do that for os * netdata/packaging/ci: test slack * netdata/packaging/ci: Bring the modified slack script on this branch * netdata/packaging/ci: Add a few more slack messages * Fix empty stage occurrence * netdata/packaging/ci: Another set of restructure changes, explained below 1) Publish release -> Publish for release (minor rename for clarity) 2) Add some more work on the make dist testing (temp folder for root and for now just validate the existence of the tarball) 3) Add some more slack notifications (that is still beta, will have to properly refine once we are done with changes of the flow) 4) Restructure the release process, finally a) relase.sh becomes draft_release.sh and changelog_and_tag_release.sh, basically splitting up the steps into meaningful independent blocks b) Adjust the pipeline, now we run 'changelog_and_tag_release.sh' to get the latest changelog and tag the release, then we run artifacts submission and draft release generation on github still more things to re-check on this step, it will become clearer once i get it up to speed to run on my local repos end to end * netdata/packaging/ci: Fix wrong path usage * netdata/packaging/ci: Update webhook URL * netdata/packaging/ci: Update stage config as proposed by the fix from travis team * netdata/packaging/ci: Another round of restructuring, after 1-1 comparison with old version Briefly: 1) Publish for release stage will only run when there is a TAG defined (To be tested though, but thats the idea). The code previously had the releaser.sh script that did exactly that, if tagger script failed, it would stop process. Now we have distributed the processing in clear chunks and we pick up the process based on shared info that travis or the environment provides 2) Rename <<Create release (only on special commit msg)>> to a more appropriate name, we are not really creating release there its more complicated 3) Move the labeler execution to the packaging step, since publishing now not happening all the time. Previously we hacked this by exit 0 with the tagger and allow the labeler (which was the second part of the execution) to run. 4) before_install in the publishing stage is wrong. We only need this as before_deploy before deploy execution. That is where we use it, really. 5) Rename changelog_and_tag_release.sh to generate_changelog_and_tag_release.sh, just to be clearer. No problem with slightly longer file names 6) Do not let generate_changelog_and_tag_release.sh execute if GIT_TAG is not defined, just exit quietly with a message and a success code 7) Remove the tagger from the generate_changelog_and_tag_release.sh, its confusing to shove it in there. Pull it out and let it run on a clear step from the pipeline but within the same machine to make sure it shares environment with generate_changelog_and_tag_release.sh (So it has to be within the same job) This is it mostly, keep in mind that the nightly and release parts are not tested yet. After this commit, there will be more code that will allow me to execute all the flows on my pipeline and fully test and adjust the workflows as needed * netdata/packaging/ci: Add more notifications, adjust the existing ones too * netdata/packaging/ci: Add safeguards for beta, do a rename too 1) rename generate_changelog.sh -> generate_changelog_for_nightlies.sh, just to provide clarity on the usage. Note: we will simplify this later, this was a workaround that raised a bug on release after refactoring nightlies. We will have to clean this up properly once we merge the new pipeline flow 2) Add checks based on TRAVIS_REPO_SLUG to identify when we running on personal repo and abort mission critical actions like deploying or generating changelogs Note: The deployment step for now will simply fail, due to the lack of the necessary encryption keys on the environment. At a later iteration, we should have beta buckets on GCS and also beta repositories on Docker to properly test our publishing capability on our beta stack * netdata/packaging/ci: Refactor README.md -- still half way there, will resume when i am done with testing * netdata/packaging/ci: At this stage make the scripts harder to run, add checks to bypass tested execution. * netdata/packaging/ci: Seems that adding the install tag within the stage, kind of overriding the outer install pull out install items from packaging to the main install, no point to isolate two packages, just install all required all the time * netdata/packaging/ci: tag obviously cant work at that step, moving it down to the jobs and i need to recheck if the syntax is applicable for travis too * netdata/packaging/ci: So the jobs in a list dont share the environment, lets see if we do it otherwise. also remove the tag check, not work as expected * netdata/packaging/ci: Add another way to check the RC condition * netdata/packaging/ci: experimental - check conditions on message * netdata/packaging/ci: Introduce an alternate master branch for the workflow testing 1. I created pipeline-redesign-master that will be the master branch for my use cases 2. Modified travis to trigger all production workflows on that branch, so that i can experiment freely with test commits to trigger process and verify execution flow * remove spaces * netdata/packaging/ci: comment out * netdata/packaging/ci: wording nit * netdata/packaging/ci: adjust the condition properly * netdata/packaging/ci: test condition for triggering on specific keywords * netdata/packaging/ci: try different syntax * netdata/packaging/ci: remove depth, causes problems as we dont get all required objects from repo (adds --depth 50 in clone) * Missing equal sign in regex? * netdata/packaging/ci: Update tagger script 1) More verbose messages 2) Update comments copyrights etc 3) Rename release_candidate to set_tag_release_candidate 4) Add curlies on the variables, also change the way we check for path * netdata/packaging/ci: reinstate git depth * netdata/packaging/ci: Remove git depth from packaging too * netdata/packaging/ci: RTFM - was using wrong syntax for slack @here approach * netdata/packaging/ci: Using install tag at the stage layer, overrides the global one for some reason. fix that * netdata/packaging/ci: Dont use the pattern matching from travis, do it outside with a script * netdata/packaging/ci: Do it the other way, script execution doesnt work there * netdata/packaging/ci: Attempt to fix conditional on release, remove the faulty assumptions as provided by travis team * netdata/packaging/ci: reintroduce the tag checking * netdata/packaging/ci: you need to bring tag checking at the stage, as it is calculated during packaging step * netdata/packaging/ci: you cant escape that one for the moment, merge them and we will rework it later again * netdata/packaging/ci: dont forget git depth * netdata/packaging/ci: [WIP] Redesign packaging release * netdata/packaging/ci: Add explicitly os definition, also remove sourcing of slack, its already done and rename before_install -> install * netdata/packaging/ci: fix stray stage name * netdata/packaging/ci: Try removing this, doesnt seem to make much sense to be there (for now) * Revert "netdata/packaging/ci: Try removing this, doesnt seem to make much sense to be there (for now)" * netdata/packaging/ci: Reinstate original design - the problem i am trying to solve is more travis-related and fundamental on the way travis works * netdata/packaging/ci: Fix broken items * netdata/packaging/ci: docker images are built and published independently. As long as tagging has taken place earlier, during packaging * netdata/packaging/ci: Following hints from https://github.com/travis-ci/travis-ci/issues/4681 and the mayhem triggered from this change So we adding the compiler, to see if it will get travis to build the right matrix. then linter broke on addons for some weird reason, so remove addons->apt->update too * netdata/packaging/ci: Now do the rest of the changes, since with addition of compiler, things got worse * netdata/packaging/ci: Ok, as expected the workaround from them did not work. Lets get creative though and assume they exclude what they added, so lets do that for os * netdata/packaging/ci: test slack * netdata/packaging/ci: Bring the modified slack script on this branch * netdata/packaging/ci: Add a few more slack messages * Fix empty stage occurrence * netdata/packaging/ci: Another set of restructure changes, explained below 1) Publish release -> Publish for release (minor rename for clarity) 2) Add some more work on the make dist testing (temp folder for root and for now just validate the existence of the tarball) 3) Add some more slack notifications (that is still beta, will have to properly refine once we are done with changes of the flow) 4) Restructure the release process, finally a) relase.sh becomes draft_release.sh and changelog_and_tag_release.sh, basically splitting up the steps into meaningful independent blocks b) Adjust the pipeline, now we run 'changelog_and_tag_release.sh' to get the latest changelog and tag the release, then we run artifacts submission and draft release generation on github still more things to re-check on this step, it will become clearer once i get it up to speed to run on my local repos end to end * netdata/packaging/ci: Fix wrong path usage * netdata/packaging/ci: Update webhook URL * netdata/packaging/ci: Update stage config as proposed by the fix from travis team * netdata/packaging/ci: Another round of restructuring, after 1-1 comparison with old version Briefly: 1) Publish for release stage will only run when there is a TAG defined (To be tested though, but thats the idea). The code previously had the releaser.sh script that did exactly that, if tagger script failed, it would stop process. Now we have distributed the processing in clear chunks and we pick up the process based on shared info that travis or the environment provides 2) Rename <<Create release (only on special commit msg)>> to a more appropriate name, we are not really creating release there its more complicated 3) Move the labeler execution to the packaging step, since publishing now not happening all the time. Previously we hacked this by exit 0 with the tagger and allow the labeler (which was the second part of the execution) to run. 4) before_install in the publishing stage is wrong. We only need this as before_deploy before deploy execution. That is where we use it, really. 5) Rename changelog_and_tag_release.sh to generate_changelog_and_tag_release.sh, just to be clearer. No problem with slightly longer file names 6) Do not let generate_changelog_and_tag_release.sh execute if GIT_TAG is not defined, just exit quietly with a message and a success code 7) Remove the tagger from the generate_changelog_and_tag_release.sh, its confusing to shove it in there. Pull it out and let it run on a clear step from the pipeline but within the same machine to make sure it shares environment with generate_changelog_and_tag_release.sh (So it has to be within the same job) This is it mostly, keep in mind that the nightly and release parts are not tested yet. After this commit, there will be more code that will allow me to execute all the flows on my pipeline and fully test and adjust the workflows as needed * netdata/packaging/ci: Add more notifications, adjust the existing ones too * netdata/packaging/ci: Add safeguards for beta, do a rename too 1) rename generate_changelog.sh -> generate_changelog_for_nightlies.sh, just to provide clarity on the usage. Note: we will simplify this later, this was a workaround that raised a bug on release after refactoring nightlies. We will have to clean this up properly once we merge the new pipeline flow 2) Add checks based on TRAVIS_REPO_SLUG to identify when we running on personal repo and abort mission critical actions like deploying or generating changelogs Note: The deployment step for now will simply fail, due to the lack of the necessary encryption keys on the environment. At a later iteration, we should have beta buckets on GCS and also beta repositories on Docker to properly test our publishing capability on our beta stack * netdata/packaging/ci: Refactor README.md -- still half way there, will resume when i am done with testing * netdata/packaging/ci: At this stage make the scripts harder to run, add checks to bypass tested execution. * netdata/packaging/ci: Seems that adding the install tag within the stage, kind of overriding the outer install pull out install items from packaging to the main install, no point to isolate two packages, just install all required all the time * netdata/packaging/ci: tag obviously cant work at that step, moving it down to the jobs and i need to recheck if the syntax is applicable for travis too * netdata/packaging/ci: So the jobs in a list dont share the environment, lets see if we do it otherwise. also remove the tag check, not work as expected * netdata/packaging/ci: Add another way to check the RC condition * netdata/packaging/ci: experimental - check conditions on message * netdata/packaging/ci: Introduce an alternate master branch for the workflow testing 1. I created pipeline-redesign-master that will be the master branch for my use cases 2. Modified travis to trigger all production workflows on that branch, so that i can experiment freely with test commits to trigger process and verify execution flow * remove spaces * netdata/packaging/ci: comment out * netdata/packaging/ci: wording nit * netdata/packaging/ci: adjust the condition properly * netdata/packaging/ci: test condition for triggering on specific keywords * netdata/packaging/ci: try different syntax * netdata/packaging/ci: remove depth, causes problems as we dont get all required objects from repo (adds --depth 50 in clone) * Missing equal sign in regex? * netdata/packaging/ci: Update tagger script 1) More verbose messages 2) Update comments copyrights etc 3) Rename release_candidate to set_tag_release_candidate 4) Add curlies on the variables, also change the way we check for path * netdata/packaging/ci: reinstate git depth * netdata/packaging/ci: Remove git depth from packaging too * netdata/packaging/ci: RTFM - was using wrong syntax for slack @here approach * netdata/packaging/ci: Using install tag at the stage layer, overrides the global one for some reason. fix that * netdata/packaging/ci: Dont use the pattern matching from travis, do it outside with a script * netdata/packaging/ci: Do it the other way, script execution doesnt work there * netdata/packaging/ci: Attempt to fix conditional on release, remove the faulty assumptions as provided by travis team * netdata/packaging/ci: reintroduce the tag checking * netdata/packaging/ci: you need to bring tag checking at the stage, as it is calculated during packaging step * netdata/packaging/ci: you cant escape that one for the moment, merge them and we will rework it later again * netdata/packaging/ci: dont forget git depth * netdata/packaging/ci: bring back the dist checking * netdata/packaging/ci: fix syntax * netdata/packaging/ci: no depth * netdata/packaging/ci: Introduce make dist validation tasks: Install from dist file and verify service is starting up * netdata/packaging/ci: Add a couple more information for troubleshooting * netdata/packaging/ci: Add extra information prior to executing crucial operations * netdata/packaging/ci: Make rm error soft here * netdata/packaging/ci: This is not needed here * netdata/packaging/ci: Prepare for beta stage deployment process - introduce a beta provider - adjust key decryption process, different keys for beta, different for prod - Enter conditions to deploy beta only on beta process and production only on production process - Use travis variables, to allow multiple users have beta stages for deployment, if they want * netdata/packaging/ci: add one more message * netdata/packaging/ci: Bring back naming to the actual master branch * netdata/packaging/ci: rename beta to dev * netdata/packaging/ci: Make permanent restrictions for some of the stricts, so that we don't mess things up * netdata/packaging/ci: Update README that was missed and also make sure git depth false is at the places needed * netdata/packaging/ci: fix codacy complaints

Paul Emm. Katsoulakis committed Apr 18, 2019 at 07:55 UTC a8ba6234f6b8540be45dc34bde4d1078a985cc22

12 files changed +609 -295

.travis.yml

+248 -86

@@ -1,113 +1,275 @@

1 -dist: trusty

1 sudo: true

2 language: c

3 services:

4 - docker

6 +# This is a hook to help us introduce "soft" errors on our process

7 +matrix:

8 + allow_failures:

9 + - env: ALLOW_SOFT_FAILURE_HERE=true

10 +

11 +# Install dependencies for all, once

12 +#

13 +install:

14 +- sudo apt-get install -y libcap2-bin zlib1g-dev uuid-dev fakeroot libipmimonitoring-dev libmnl-dev libnetfilter-acct-dev gnupg python-pip

15 +- sudo apt install -y --only-upgrade docker-ce

16 +- sudo pip install git-semver

17 +- docker info

18 +- source tests/installer/slack.sh

19 +

20 +# Setup notification system

21 +#

22 +notifications:

23 + webhooks: https://app.fossa.io/hooks/travisci

24 +

25 +# Define the stage sequence and conditionals

26 +#

27 stages:

8 -- test

9 -- build

10 -- name: packaging

11 - if: branch = master AND type != pull_request AND type != cron

12 -- name: nightlies

28 +# Mandatory runs, we always want these executed

29 +- name: Code quality, linting, syntax, code style

30 +- name: Build process

31 +- name: Artifacts validation

32 +

33 + # Nightly operations

34 +- name: Nightly operations

35 if: branch = master AND type = cron

14 -- name: Integrity testing

36 +- name: Nightly release

37 if: branch = master AND type = cron

39 + # Scheduled releases

40 +- name: Packaging for release

41 + if: branch = master AND type != pull_request AND type != cron

42 +

43 +- name: Publish for release

44 + if: branch = master AND type != pull_request AND type != cron AND commit_message =~ /(\[netdata release candidate\]|\[netdata major release\]|\[netdata minor release\]|\[netdata patch release\])/

45 +

46 +

47 +

48 +# Define stage implementation details

49 +#

50 jobs:

51 include:

19 - - stage: test

20 - name: C

21 - install: sudo apt-get install -y libcap2-bin zlib1g-dev uuid-dev fakeroot libipmimonitoring-dev libmnl-dev libnetfilter-acct-dev

22 - script: fakeroot ./netdata-installer.sh --install $HOME --dont-wait --dont-start-it --enable-plugin-nfacct --enable-plugin-freeipmi --disable-lto && $HOME/netdata/usr/sbin/netdata -W unittest

23 - env: CFLAGS='-O1 -DNETDATA_INTERNAL_CHECKS=1 -DNETDATA_VERIFY_LOCKS=1'

24 - - name: dashboard.js

25 - script: cp web/gui/dashboard.js /tmp/dashboard.js && ./build/build.sh && diff /tmp/dashboard.js web/gui/dashboard.js

26 - - name: lint .sh.in files

52 + # Do code quality, syntax checking and other pre-build activities

53 + - stage: Code quality, linting, syntax, code style

54 +

55 + name: Run shellchecking on BASH

56 script: shellcheck --format=gcc $(find . -name '*.sh.in' -not -iwholename '*.git*')

28 - - name: check checksums for kickstart files

57 +

58 + # This falls under same stage defined earlier

59 + - name: Run checksum checks on kickstart files

60 script: ./tests/installer/checksums.sh

61 env: LOCAL_ONLY="true"

31 - - name: Kickstart Script execution integrity

32 - script: docker run -it -v "${PWD}:/code:rw" -w /code "netdata/os-test:centos7" packaging/installer/kickstart.sh --dont-wait --dont-start-it --install /tmp

33 - - name: Kickstart-static64 Script execution integrity

34 - before_script: rm -rf /tmp/kickstart-static64-temp && mkdir -p /tmp/kickstart-static64-temp

35 - script: docker run -it -v "${PWD}:/code:rw" -v "/tmp/kickstart-static64-temp:/opt" -w /code "netdata/os-test:centos7" packaging/installer/kickstart-static64.sh --dont-wait --dont-start-it

36 - after_script: rm -rf /tmp/kickstart-static64-temp

37 - - name: coverity

38 - install: sudo apt-get install -y zlib1g-dev uuid-dev libipmimonitoring-dev libmnl-dev libnetfilter-acct-dev

39 - script: ./coverity-install.sh && ./coverity-scan.sh || echo "Coverity failed :("

40 - if: type = cron

41 -

42 - - stage: build

43 -# TODO: enable when travis OSX become stable. Probably after 12.01.2019

44 -# name: OSX

45 -# install: brew install fakeroot ossp-uuid

46 -# script: fakeroot ./netdata-installer.sh --install $HOME --dont-wait --dont-start-it

47 -# os: osx

48 -# - name: ubuntu 14.04 (not containerized)

49 - name: ubuntu 14.04 (not containerized)

50 - install: sudo apt-get install -y libcap2-bin zlib1g-dev uuid-dev fakeroot

51 - script: fakeroot ./netdata-installer.sh --dont-wait --dont-start-it --install $HOME

52 - - name: build container (alpine installation)

53 - script: ./packaging/docker/build.sh

62 +

63 + # This falls under same stage defined earlier

64 + - name: Web Dashboard pre-generated file consistency checks (dashboard.js)

65 + script: cp web/gui/dashboard.js /tmp/dashboard.js && ./build/build.sh && diff /tmp/dashboard.js web/gui/dashboard.js

66 +

67 +

68 +

69 + # Ensure netdata code builds successfully

70 + - stage: Build process

71 +

72 + name: Standard netdata build

73 + script: fakeroot ./netdata-installer.sh --install $HOME --dont-wait --dont-start-it --enable-plugin-nfacct --enable-plugin-freeipmi --disable-lto

74 + env: CFLAGS='-O1 -DNETDATA_INTERNAL_CHECKS=1 -DNETDATA_VERIFY_LOCKS=1'

75 + after_failure: post_message "TRAVIS_MESSAGE" "<!here> standard netdata build is failing (Still dont know which one, will improve soon)"

76 +

77 + - name: Docker container build process (alpine installation)

78 + script: packaging/docker/build.sh

79 env: DEVEL="true"

55 - - name: ubuntu 18.04 + lifecycle

80 + after_failure: post_message "TRAVIS_MESSAGE" "Docker build process failed"

81 +

82 + - name: Run 'make dist' validation

83 + before_script: mkdir /tmp/netdata-makedist-test

84 + script:

85 + - echo "GIT Branch:" && git branch

86 + - echo "Last commit:" && git log -1

87 + - echo "GIT Describe:" && git describe

88 + - echo "packaging/version:" && cat packaging/version

89 + - docker run -it -v "${PWD}:/code:rw" -v "/tmp/netdata-makedist-test:/netdata_install:rw" -w /code "netdata/os-test:ubuntu1804" make clean || echo "Nothing to clean"

90 + - docker run -it -v "${PWD}:/code:rw" -v "/tmp/netdata-makedist-test:/netdata_install:rw" -w /code "netdata/os-test:ubuntu1804" make distclean || echo "Nothing to distclean"

91 + - docker run -it -v "${PWD}:/code:rw" -v "/tmp/netdata-makedist-test:/netdata_install:rw" -w /code "netdata/os-test:ubuntu1804" autoreconf -ivf && ./configure --prefix=/netdata_install/usr --sysconfdir=/netdata_install/etc --localstatedir=/netdata_install/var --with-zlib --with-math --with-user=netdata CFLAGS=-O2

92 + - docker run -it -v "${PWD}:/code:rw" -v "/tmp/netdata-makedist-test:/netdata_install:rw" -w /code "netdata/os-test:ubuntu1804" make dist

93 + - docker run -it -v "${PWD}:/code:rw" -v "/tmp/netdata-makedist-test:/netdata_install:rw" -w /code "netdata/os-test:ubuntu1804" ls -ltr ./netdata-$(git describe).tar.gz || ls -ltr ./netdata-$(cat packaging/version | tr -d '\n').tar.gz

94 + - .travis/run_install_with_dist_file.sh

95 + - docker run -it -v "${PWD}:/code:rw" -v "/tmp/netdata-makedist-test:/netdata_install:rw" -w /code "netdata/os-test:ubuntu1804" make distclean

96 + git:

97 + depth: false

98 + after_script: rm -rf /tmp/netdata-makedist-test

99 + after_failure: post_message "TRAVIS_MESSAGE" "'make dist' failed"

100 +

101 +

102 +

103 + - stage: Artifacts validation

104 +

105 + name: Unit Testing

106 + script:

107 + - fakeroot ./netdata-installer.sh --install $HOME --dont-wait --dont-start-it --enable-plugin-nfacct --enable-plugin-freeipmi --disable-lto

108 + - $HOME/netdata/usr/sbin/netdata -W unittest

109 + env: CFLAGS='-O1 -DNETDATA_INTERNAL_CHECKS=1 -DNETDATA_VERIFY_LOCKS=1'

110 + after_failure: post_message "TRAVIS_MESSAGE" "Unit testing failed"

111 +

112 + - name: Build/install on ubuntu 14.04 (not containerized)

113 + script: fakeroot ./netdata-installer.sh --dont-wait --dont-start-it --install $HOME

114 + after_failure: post_message "TRAVIS_MESSAGE" "Build/Install failed on ubuntu 14.04"

115 +

116 + - name: Build/Install for ubuntu 18.04 (not containerized)

117 + script: fakeroot ./netdata-installer.sh --dont-wait --dont-start-it --install $HOME

118 + after_failure: post_message "TRAVIS_MESSAGE" "Build/Install failed on ubuntu 18.04"

119 +

120 + - name: Run netdata lifecycle, on ubuntu 18.04 (Containerized)

121 script: docker run -it -v "${PWD}:/code:rw" -w /code "netdata/os-test:ubuntu1804" bats --tap tests/lifecycle.bats

57 - - name: CentOS 7

58 - script: docker run -it -v "${PWD}:/code:rw" -w /code "netdata/os-test:centos7" ./netdata-installer.sh --dont-wait --dont-start-it --install /tmp

59 - - name: CentOS 6

122 + after_failure: post_message "TRAVIS_MESSAGE" "Netdata lifecycle test script failed on ubuntu 18.04"

123 +

124 + - name: Build/install for CentOS 6 (Containerized)

125 script: docker run -it -v "${PWD}:/code:rw" -w /code "netdata/os-test:centos6" ./netdata-installer.sh --dont-wait --dont-start-it --install /tmp

126 + after_failure: post_message "TRAVIS_MESSAGE" "Build/Install failed on CentOS 6"

127

62 - - stage: packaging

63 - name: Create release (only on special commit msg)

64 - install:

65 - - sudo apt-get install -y gnupg libcap2-bin zlib1g-dev uuid-dev fakeroot python-pip

66 - - sudo apt install -y --only-upgrade docker-ce

67 - - docker info

68 - before_script: sudo pip install git-semver

69 - script: ".travis/releaser.sh && .travis/labeler.sh" # labeler should be replaced with GitHub Actions when they hit GA

128 + - name: Build/install for CentOS 7 (Containerized)

129 + script: docker run -it -v "${PWD}:/code:rw" -w /code "netdata/os-test:centos7" ./netdata-installer.sh --dont-wait --dont-start-it --install /tmp

130 + after_failure: post_message "TRAVIS_MESSAGE" "Build/Install failed on CentOS 7"

131 +

132 +

133 +

134 + - stage: Packaging for release

135 +

136 + name: Generate changelog and TAG the release (only on special commit msg)

137 + before_script: post_message "TRAVIS_MESSAGE" "Packaging step for release initiated"

138 + script: .travis/generate_changelog_and_tag_release.sh

139 + after_failure: post_message "TRAVIS_MESSAGE" "<!here> Packaging for release failed"

140 git:

141 depth: false

142

73 - - stage: nightlies

74 - name: Nightly build

75 - before_install: openssl aes-256-cbc -K $encrypted_8daf19481253_key -iv $encrypted_8daf19481253_iv -in .travis/gcs-credentials.json.enc -out .travis/gcs-credentials.json -d

76 - install:

77 - - sudo apt-get install -y gnupg libcap2-bin zlib1g-dev uuid-dev fakeroot

78 - - sudo apt install -y --only-upgrade docker-ce

79 - - docker info

80 - script: ".travis/nightlies.sh"

143 + - name: Run labeler on github issues

144 + script: .travis/labeler.sh # labeler should be replaced with GitHub Actions when they hit GA

145 +

146 +

147 +

148 + # We only publish if a TAG has been set during packaging

149 + - stage: Publish for release

150 +

151 + name: Build & Publish docker images

152 + before_script: post_message "TRAVIS_MESSAGE" "Publishing docker images"

153 + script:

154 + - echo "GIT Branch:" && git branch

155 + - echo "Last commit:" && git log -1

156 + - echo "GIT Describe:" && git describe

157 + - echo "packaging/version:" && cat packaging/version

158 + - packaging/docker/build.sh

159 + - packaging/docker/publish.sh

160 + after_failure: post_message "TRAVIS_MESSAGE" "<!here> Docker image publishing failed"

161 git:

162 depth: false

83 - deploy:

84 - provider: gcs

85 - edge:

86 - branch: gcs-ng

87 - project_id: netdata-storage

88 - credentials: .travis/gcs-credentials.json

89 - bucket: "netdata-nightlies"

90 - skip_cleanup: true

91 - local_dir: "artifacts"

92 - on:

93 - # Only deploy on netdata/netdata, master branch, when artifacts directory is created

94 - repo: netdata/netdata

95 - branch: master

96 - condition: -d "artifacts"

97 - after_deploy: rm -f .travis/gcs-credentials.json

163 + env: ALLOW_SOFT_FAILURE_HERE=true

164 + # We don't run on release candidates

165 + if: tag !~ /(-rc)/

166 +

167 + - name: Generate release artifacts

168 + before_script: post_message "TRAVIS_MESSAGE" "Starting artifacts generation for release"

169 + script: .travis/create_artifacts.sh

170 + after_failure: post_message "TRAVIS_MESSAGE" "<!here> Release artifacts generation failed"

171 + git:

172 + depth: false

173 + # We don't run on release candidates

174 + if: tag !~ /(-rc)/

175 +

176 + - name: Create release draft

177 + before_script: post_message "TRAVIS_MESSAGE" "Drafting release on github"

178 + script:

179 + - echo "GIT Branch:" && git branch

180 + - echo "Last commit:" && git log -1

181 + - echo "GIT Describe:" && git describe

182 + - echo "packaging/version:" && cat packaging/version

183 + - .travis/draft_release.sh

184 + git:

185 + depth: false

186 + after_failure: post_message "TRAVIS_MESSAGE" "<!here> Draft release submission failed"

187 + # We don't run on release candidates

188 + if: tag !~ /(-rc)/

189 +

190 +

191 +

192 + # This is the nightly pre-execution step (Jobs, preparatory steps for nightly, etc)

193 + - stage: Nightly operations

194 +

195 + name: Run coverity scan

196 + # Just notify people that Nightly ops triggered, use the first step as a hook to do that

197 + before_script: post_message "TRAVIS_MESSAGE" "Starting nightly operations"

198 + script: ./coverity-install.sh && ./coverity-scan.sh || echo "Coverity failed :("

199

99 - - stage: Integrity testing

100 - name: Kickstart Files integrity testing

200 + - name: Kickstart files integrity testing (extended)

201 script: ./tests/installer/checksums.sh

102 - - name: Kickstart Execution integrity

103 - before_script: rm -rf /tmp/kickstart-static64-temp && mkdir -p /tmp/kickstart-static64-temp

202 +

203 + - name: Run labeler on github issues

204 + script: .travis/labeler.sh # labeler should be replaced with GitHub Actions when they hit GA

205 +

206 + # This is generating the changelog for nightly release and publish it

207 + - name: Generate nightly changelog

208 + before_script: post_message "TRAVIS_MESSAGE" "Starting changelog generation for nightlies"

209 + script: ".travis/nightlies.sh"

210 + after_failure: post_message "TRAVIS_MESSAGE" "<!here> Nightly changelog generation failed"

211 + git:

212 + depth: false

213 +

214 +

215 +

216 + # This is the nightly execution step

217 + #

218 + - stage: Nightly release

219 +

220 + name: Build & Publish docker images

221 + before_script: post_message "TRAVIS_MESSAGE" "Publishing docker images for nightlies"

222 script:

105 - - docker run -it -v "${PWD}:/code:rw" -w /code "netdata/os-test:centos7" packaging/installer/kickstart.sh --dont-wait --dont-start-it --install /tmp

106 - - docker run -it -v "${PWD}:/code:rw" -v "/tmp/kickstart-static64-temp:/opt" -w /code "netdata/os-test:centos7" packaging/installer/kickstart-static64.sh --dont-wait --dont-start-it

107 - after_script: rm -rf /tmp/kickstart-static64-temp

223 + - docker info

224 + - packaging/docker/build.sh

225 + - packaging/docker/publish.sh

226 + after_failure: post_message "TRAVIS_MESSAGE" "<!here> Nightly docker image publish failed"

227 + env: ALLOW_SOFT_FAILURE_HERE=true

228

109 -notifications:

110 - webhooks: https://app.fossa.io/hooks/travisci

111 - slack:

112 - rooms:

113 - - secure: "NuW1p7s+WGLcyhEceeiLRSV1JgAc6N47zgdSsYoxrjSFRQHDfc8jensypDcEJwgs1K2Hcve9iKRaAddEHEw7AkS6rie9gFR5HmmbKXfW2GFMqOr6maYTFsvaECPqiWk1n9/XnRLsAi5kZ8HxH+a8ldb/eaVoFQesY1jPXgh11BM5DwvpXjEtwg0WGASsKiymvnXFS3KcC+sR7Lln2GX1a8vfCX2I3TEmOedKMlSHUy5JilGGC3AWA0SWS8tR8PUH0u3dHL5j0RNIr1RO3Yx24QgUpg/YpvKymnW/iIIEOq2vb2mBhhiKEQjJ1djUL4VSPzjIDpUzThVpKaHk3syOp6W9qZEHKhR/sqjc5Yk2XRjsw1cM0nS60gaCgxtKhEMKWcjtvWf04oJAVrmcUwcYXj0eA+jgRCZl5VhyufK/fUJavjOfsQGjwhdjxQfwDCw33W17ypJUt4GZngdb6jbIhEOcKHSLQDu1vuHTw82hJJkthkmR59PX30qJdl/MEGcfVLdN/fkCokjR/qwfmkNwQm+wYSKsK/Jq4RgBT0/oZwY3e8nkCq2ov7lBbDO3/0rzQKWZ9Uy//tnoCM3vGhDwGHQxsHshv7g6KwdhYTcmm7WWWIucfLupcjFUO1HbRuJ+7ZnvxRRwKiV+MGkFT2SNJkS8q1/jCu9KGbmktd0WUSE="

229 + - name: Create nightly release artifacts, publish to GCS

230 + before_script: post_message "TRAVIS_MESSAGE" "Starting artifacts generation for nightlies"

231 + script:

232 + - echo "GIT Branch:" && git branch

233 + - echo "Last commit:" && git log -1

234 + - echo "GIT Describe:" && git describe

235 + - echo "packaging/version:" && cat packaging/version

236 + - .travis/create_artifacts.sh

237 + after_failure: post_message "TRAVIS_MESSAGE" "<!here> Nightly artifacts generation failed"

238 + before_deploy:

239 + echo "Preparing creds under ${TRAVIS_REPO_SLUG}";

240 + if [ "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then

241 + openssl aes-256-cbc -K $encrypted_8daf19481253_key -iv $encrypted_8daf19481253_iv -in .travis/gcs-credentials.json.enc -out .travis/gcs-credentials.json -d;

242 + else

243 + echo "Beta deployment stage in progress";

244 + openssl aes-256-cbc -K $encrypted_8daf19481253_key -iv $encrypted_8daf19481253_iv -in .travis/gcs-credentials.json.enc -out .travis/gcs-credentials.json -d;

245 + fi;

246 + deploy:

247 + # Beta storage, used for testing purposes

248 + - provider: gcs

249 + edge:

250 + branch: gcs-ng

251 + project_id: netdata-storage

252 + credentials: .travis/gcs-credentials.json

253 + bucket: "netdata-dev-nightlies"

254 + skip_cleanup: true

255 + local_dir: "artifacts"

256 + on:

257 + # Only deploy on netdata/netdata, master branch, when artifacts directory is created

258 + repo: ${TRAVIS_REPO_SLUG}

259 + branch: master

260 + condition: -d "artifacts"

261 + # Production storage

262 + - provider: gcs

263 + edge:

264 + branch: gcs-ng

265 + project_id: netdata-storage

266 + credentials: .travis/gcs-credentials.json

267 + bucket: "netdata-nightlies"

268 + skip_cleanup: true

269 + local_dir: "artifacts"

270 + on:

271 + # Only deploy on netdata/netdata, master branch, when artifacts directory is created

272 + repo: netdata/netdata

273 + branch: master

274 + condition: -d "artifacts"

275 + after_deploy: rm -f .travis/gcs-credentials.json

.travis/README.md

+87 -81

@@ -8,84 +8,90 @@

8 - encrypted_8daf19481253_key - key needed by openssl to decrypt GCS credentials file

9 - encrypted_8daf19481253_iv - IV needed by openssl to decrypt GCS credentials file

10 - COVERITY_SCAN_TOKEN - Token to allow coverity test analysis uploads

11 -

12 -## Stages

13 -

14 -### Test

15 -

16 -Unit tests and coverage tests are executed here. Stage consists of 2 parallel jobs:

17 - - C tests - executed every time

18 - - dashboard.js - test if source files create the same file as it is in current repo

19 - - coverity test - executed only when pipeline was triggered from cron

20 -

21 -### Build

22 -

23 -Stage is executed every time and consists of 5 parallel jobs which execute containerized and non-containerized

24 -installations of netdata. Jobs are run on following operating systems:

25 - - OSX

26 - - ubuntu 14.04

27 - - ubuntu 16.04 (containerized)

28 - - CentOS 6 (containerized)

29 - - CentOS 7 (containerized)

30 - - alpine (containerized)

31 -

32 -Images for system containers are stored on dockerhub and are created from Dockerfiles located in

33 -[netdata/helper-images](https://github.com/netdata/helper-images) repository.

34 -

35 -### Packaging

36 -

37 -This stage is executed only on "master" brach and allows us to create a new tag just looking at git commit message.

38 -It executes one script called `releaser.sh` which is responsible for creating a release on GitHub by using

39 -[hub](https://github.com/github/hub). This script is also executing other scripts which can also be used in other

40 -CI jobs:

41 - - `.travis/tagger.sh`

42 - - `.travis/generate_changelog.sh`

43 - - `packaging/docker/build.sh`

44 - - `.travis/create_artifacts.sh`

45 -

46 -Alternatively new release can be also created by pushing new tag to master branch.

47 -Additionally this step is also executing `.travis/labeler.sh` which is a temporary workaround to automatically label

48 -issues and PR. This script should be replaced with GitHub Actions when they are available to public.

49 -

50 -##### tagger.sh

51 -

52 -Script responsible to find out what will be the next tag based on a keyword in last commit message. Keywords are:

53 - - `[netdata patch release]` to bump patch number

54 - - `[netdata minor release]` to bump minor number

55 - - `[netdata major release]` to bump major number

56 - - `[netdata release candidate]` to create a new release candidate (appends or modifies suffix `-rcX` of previous tag)

57 -All keywords MUST be surrounded with square brackets.

58 -Tag is then stored in `GIT_TAG` variable.

59 -

60 -##### generate_changelog.sh

61 -

62 -Automatic changelog generator which updates our CHANGELOG.md file based on GitHub features (mostly labels and pull

63 -requests). Internally it uses

64 -[github-changelog-generator](https://github.com/github-changelog-generator/github-changelog-generator) and more

65 -information can be found on that project site.

66 -

67 -##### build.sh and create_artifacts.sh

68 -

69 -Scripts used to build new container images and provide release artifacts (tar.gz and makeself archives)

70 -

71 -### Nightlies

72 -

73 -##### Tarball and self-extractor build AND Nightly docker images

74 -

75 -As names might suggest those two jobs are responsible for nightly netdata package creation and are run every day (in

76 -cron). Combined they produce:

77 - - docker images

78 - - tar.gz archive (soon to be removed)

79 - - self-extracting package

80 -

81 -This is achieved by running 2 scripts described earlier:

82 - - `create_artifacts.sh`

83 - - `build.sh`

84 -

85 -Artifacts are pushed to GCS and container images are stored in docker hub.

86 -

87 -##### Changelog generation

88 -

89 -This job is responsible for regenerating changelog every day by executing `generate_changelog.sh` script. This is done

90 -only once a day due to github rate limiter.

91 -

11 +- SLACK_USERNAME - This is required for the slack notifications triggered by travis pipeline

12 +- SLACK_CHANNEL - This is the channel that Travis will be posting messages

13 +- SLACK_NOTIFY_WEBHOOK_URL - This is the incoming URL webhook as provided by slack integration. Visit Apps integration in slack to generate the required hook

14 +- SLACK_BOT_NAME - This is the name your bot will appear with on slack

15 +

16 +## CI workflow details

17 +Our CI pipeline is designed to help us identify and mitigate risks at all stages of implementation.

18 +To accommodate this need, we used [Travis CI](http://www.travis-ci.com) as our CI/CD tool.

19 +Our main areas of concern are:

20 +1) Only push code that is working. That means fail fast so that we can improve before we reach the public

21 +

22 +2) Reduce the time to market to minimum, by streamlining the release process.

23 + That means a lot of testing, a lot of consistency checks, a lot of validations

24 +

25 +3) Generated artifacts consistency. We should not allow broken software to reach the public.

26 + When this happens, it's embarassing and we struggle to eliminate it.

27 +

28 +4) We are an innovative company, so we love to automate :)

29 +

30 +

31 +Having said that, here's a brief introduction to Netdata's improved CI/CD pipeline with Travis.

32 +Our CI/CD lifecycle contains three different execution entry points:

33 +1) A user opens a pull request to netdata/master: Travis will run a pipeline on the branch under that PR

34 +2) A merge or commit happens on netdata/master. This will trigger travis to run, but we have two distinct cases in this scenario:

35 + a) A user merges a pull request to netdata/master: Travis will run on master, after the merge.

36 + b) A user runs a commit/merge with a special keyword (mentioned later).

37 + This triggers a release for either minor, major or release candidate versions, depending the keyword

38 +3) A scheduled job runs on master once per day: Travis will run on master at the scheduled interval

39 +

40 +To accommodate all three entry points our CI/CD workflow has a set of steps that run on all three entry points.

41 +Once all these steps are successfull, then our pipeline executes another subset of steps for entry points 2 and 3.

42 +In travis terms the "steps" are "Stages" and within each stage we execute a set of activities called "jobs" in travis.

43 +

44 +### Always run: Stages that running on all three execution entry points

45 +

46 +## Code quality, linting, syntax, code style

47 +At this early stage we iterate through a set of basic quality control checks:

48 +- Shell checking: Run linters for our various BASH scripts

49 +- Checksum validators: Run validators to ensure our installers and documentation are in sync

50 +- Dashboard validator: We provide a pre-generated dashboard.js script file that we need to make sure its up to date. We validate that.

51 +

52 +## Build process

53 +At this stage, basically, we build :-)

54 +We do a baseline check of our build artifacts to guarantee they are not broken

55 +Briefly our activities include:

56 +- Verify docker builds successfully

57 +- Run the standard netdata installer, to make sure we build & run properly

58 +- Do the same through 'make dist', as this is our stable channel for our kickstart files

59 +

60 +## Artifacts validation

61 +At this point we know our software is building, we need to go through the a set of checks, to guarantee

62 +that our product meets certain epxectations. At the current stage, we are focusing on basic capabilities

63 +like installing in different distributions, running the full lifecycle of install-run-update-install and so on.

64 +We are still working on enriching this with more and more use cases, to get us closer to achieving full stability of our software.

65 +Briefly we currently evaluate the following activities:

66 +- Basic software unit testing

67 +- Non containerized build and install on ubuntu 14.04

68 +- Non containerized build and install on ubuntu 18.04

69 +- Running the full netdata lifecycle (install, update, uninstall) on ubuntu 18.04

70 +- Build and install on CentOS 6

71 +- Build and install on CentOS 7

72 +(More to come)

73 +

74 +### Nightly operations: Stages that run daily under cronjob

75 +The nightly stages are related to the daily nightly activities, that produce our daily latest releases.

76 +We also maintain a couple of cronjobs that run during the night to provide us with deeper insights,

77 +like for example coverity scanning or extended kickstart checksum checks

78 +

79 +## Nightly operations

80 +At this stage we run scheduled jobs and execute the nightly changelog generator, coverity scans,

81 +labeler for our issues and extended kickstart files checksum validations.

82 +

83 +## Nightly release

84 +During this stage we are building and publishing latest docker images, prepare the nightly artifacts

85 +and deploy them (the artifacts) to our google cloud service provider.

86 +

87 +

88 +### Publishing

89 +Publishing is responsible for executing the major/minor/patch releases and is separated

90 +in two stages: packaging preparation process and publishing.

91 +

92 +## Packaging for release

93 +During packaging we are preparing the release changelog information and run the labeler.

94 +

95 +## Publish for release

96 +The publishing stage is the most complex part in publishing. This is the stage were we generate and publish docker images,

97 +prepare the release artifacts and get ready with the release draft.

.travis/create_artifacts.sh

@@ -8,6 +8,11 @@ if [ ! -f .gitignore ]; then

8 exit 1

9 fi

11 +if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then

12 + echo "Beta mode on ${TRAVIS_REPO_SLUG}, not running anything here"

13 + exit 0

14 +fi;

15 +

16 # Everything from this directory will be uploaded to GCS

17 mkdir -p artifacts

18 BASENAME="netdata-$(git describe)"

.travis/draft_release.sh new

+60

@@ -0,0 +1,60 @@

1 +#!/bin/bash

2 +#

3 +# Draft release generator.

4 +# This utility is responsible for submitting a draft release to github repo

5 +# It is agnostic of other processes, when executed it will draft a release,

6 +# based on the most recent reachable tag.

7 +#

8 +# Requirements:

9 +# - GITHUB_TOKEN variable set with GitHub token. Access level: repo.public_repo

10 +# - artifacts directory in place

11 +# - The directory is created by create_artifacts.sh mechanism

12 +# - The artifacts need to be created with the same tag, obviously

13 +#

14 +# Copyright: SPDX-License-Identifier: GPL-3.0-or-later

15 +#

16 +# Author: Pavlos Emm. Katsoulakis <paul@netdata.cloud>

17 +

18 +set -e

19 +

20 +if [ ! -f .gitignore ]; then

21 + echo "Run as ./travis/$(basename "$0") from top level directory of git repository"

22 + exit 1

23 +fi

24 +

25 +if [[ $(git describe) =~ -rc* ]]; then

26 + echo "This is a release candidate tag, we do not generate a release draft"

27 + exit 0

28 +fi

29 +

30 +# Load the tag, if any

31 +GIT_TAG=$(git describe)

32 +

33 +if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then

34 + echo "Beta mode on ${TRAVIS_REPO_SLUG}, i was about to run for release (${GIT_TAG}), but i am emulating, so bye"

35 + exit 0

36 +fi;

37 +

38 +echo "---- CREATING RELEASE DRAFT WITH ASSETS -----"

39 +# Download hub

40 +HUB_VERSION=${HUB_VERSION:-"2.5.1"}

41 +wget "https://github.com/github/hub/releases/download/v${HUB_VERSION}/hub-linux-amd64-${HUB_VERSION}.tgz" -O "/tmp/hub-linux-amd64-${HUB_VERSION}.tgz"

42 +tar -C /tmp -xvf "/tmp/hub-linux-amd64-${HUB_VERSION}.tgz"

43 +export PATH=$PATH:"/tmp/hub-linux-amd64-${HUB_VERSION}/bin"

44 +

45 +# Create a release draft

46 +if [ -z ${GIT_TAG+x} ]; then

47 + echo "Variable GIT_TAG is not set. Something went terribly wrong! Exiting."

48 + exit 1

49 +fi

50 +if [ "${GIT_TAG}" != "$(git tag --points-at)" ]; then

51 + echo "ERROR! Current commit is not tagged. Stopping release creation."

52 + exit 1

53 +fi

54 +until hub release create --draft \

55 + -a "artifacts/netdata-${GIT_TAG}.tar.gz" \

56 + -a "artifacts/netdata-${GIT_TAG}.gz.run" \

57 + -a "artifacts/sha256sums.txt" \

58 + -m "${GIT_TAG}" "${GIT_TAG}"; do

59 + sleep 5

60 +done

.travis/generate_changelog_and_tag_release.sh new

+69

@@ -0,0 +1,69 @@

1 +#!/bin/bash

2 +#

3 +# Script to automatically do a couple of things:

4 +# - generate a new tag according to semver (https://semver.org/)

5 +# - generate CHANGELOG.md by using https://github.com/skywinder/github-changelog-generator

6 +#

7 +# Tags are generated by searching for a keyword in last commit message. Keywords are:

8 +# - [patch] or [fix] to bump patch number

9 +# - [minor], [feature] or [feat] to bump minor number

10 +# - [major] or [breaking change] to bump major number

11 +# All keywords MUST be surrounded with square braces.

12 +#

13 +# Script uses git mechanisms for locking, so it can be used in parallel builds

14 +#

15 +# Requirements:

16 +# - GITHUB_TOKEN variable set with GitHub token. Access level: repo.public_repo

17 +# - docker

18 +#

19 +# This is a modified version of:

20 +# https://github.com/paulfantom/travis-helper/blob/master/releasing/releaser.sh

21 +#

22 +# Copyright: SPDX-License-Identifier: GPL-3.0-or-later

23 +#

24 +# Author: Pavlos Emm. Katsoulakis <paul@netdata.cloud>

25 +# Author: Pawel Krupa (@paulfantom)

26 +set -e

27 +

28 +if [ ! -f .gitignore ]; then

29 + echo "Run as ./travis/$(basename "$0") from top level directory of git repository"

30 + exit 1

31 +fi

32 +

33 +echo "--- Executing Tagging facility to determine TAG ---"

34 +.travis/tagger.sh

35 +

36 +echo "--- Changelog generator and tagger script starting ---"

37 +# If tagger script hasn't produced a TAG, there is nothing to do so bail out happy

38 +if [ -z "${GIT_TAG}" ]; then

39 + echo "GIT_TAG is empty, nothing to do for now (Value: $GIT_TAG)"

40 + exit 0

41 +fi

42 +

43 +if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then

44 + echo "Beta mode on ${TRAVIS_REPO_SLUG}, nothing to do on the changelog generator and tagging script for (${GIT_TAG}), bye"

45 + exit 0

46 +fi

47 +

48 +echo "--- Initialize git configuration ---"

49 +export GIT_MAIL="bot@netdata.cloud"

50 +export GIT_USER="netdatabot"

51 +git config user.email "${GIT_MAIL}"

52 +git config user.name "${GIT_USER}"

53 +git checkout master

54 +git pull

55 +

56 +echo "---- UPDATE VERSION FILE ----"

57 +echo "$GIT_TAG" >packaging/version

58 +git add packaging/version

59 +

60 +echo "---- GENERATE CHANGELOG -----"

61 +./.travis/generate_changelog_for_release.sh

62 +git add CHANGELOG.md

63 +

64 +echo "---- COMMIT AND PUSH CHANGES ----"

65 +git commit -m "[ci skip] release $GIT_TAG"

66 +git tag "$GIT_TAG" -a -m "Automatic tag generation for travis build no. $TRAVIS_BUILD_NUMBER"

67 +git push "https://${GITHUB_TOKEN}:@$(git config --get remote.origin.url | sed -e 's/^https:\/\///')"

68 +git push "https://${GITHUB_TOKEN}:@$(git config --get remote.origin.url | sed -e 's/^https:\/\///')" --tags

69 +# After those operations output of command `git describe` should be identical with a value of GIT_TAG

.travis/generate_changelog_for_nightlies.sh renamed

+6 -1

@@ -31,8 +31,13 @@ if [ -z ${GIT_TAG+x} ]; then

31 else

32 OPTS="--future-release ${GIT_TAG}"

33 fi

34 -

34 echo "We got $COMMITS_SINCE_RELEASE changes since $LAST_TAG, re-generating changelog"

35 +

36 +if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then

37 + echo "Beta mode on ${TRAVIS_REPO_SLUG}, nothing else to do here"

38 + exit 0

39 +fi

40 +

41 git config user.email "${GIT_MAIL}"

42 git config user.name "${GIT_USER}"

43 git checkout master

.travis/generate_changelog_for_release.sh

+5 -2

@@ -11,13 +11,17 @@ ORGANIZATION=$(echo "$TRAVIS_REPO_SLUG" | awk -F '/' '{print $1}')

11 PROJECT=$(echo "$TRAVIS_REPO_SLUG" | awk -F '/' '{print $2}')

12 GIT_MAIL=${GIT_MAIL:-"bot@netdata.cloud"}

13 GIT_USER=${GIT_USER:-"netdatabot"}

14 -

14 if [ -z ${GIT_TAG+x} ]; then

15 OPTS=""

16 else

17 OPTS="--future-release ${GIT_TAG}"

18 fi

20 +if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then

21 + echo "Beta mode on ${TRAVIS_REPO_SLUG}, nothing else to do"

22 + exit 0

23 +fi

24 +

25 echo "--- Creating changelog ---"

26 git checkout master

27 git pull

@@ -30,4 +34,3 @@ docker run -it -v "$(pwd)":/project markmandel/github-changelog-generator:latest

34 --unreleased-label "**Next release**" \

35 --exclude-labels "stale,duplicate,question,invalid,wontfix,discussion,no changelog" \

36 --no-compare-link ${OPTS}

33 -

.travis/nightlies.sh

+10 -14

@@ -1,11 +1,9 @@

1 #!/bin/bash

2 #

3 -# This is the nightlies orchastration script

4 -# It runs the following activities in order:

5 -# 1) Generate changelog

6 -# 2) Build docker images

7 -# 3) Publish docker images

8 -# 4) Generate the rest of the artifacts (Source code .tar.gz file and makeself binary generation)

3 +# This is the nightly changelog generation script

4 +# It is responsible for two major activities:

5 +# 1) Update packaging/version with the current nightly version

6 +# 2) Generate the changelog for the mentioned version

7 #

8 # Copyright: SPDX-License-Identifier: GPL-3.0-or-later

9 #

@@ -34,14 +32,12 @@ if [ "${COMMITS_SINCE_RELEASE}" == "${PREVIOUS_NIGHTLY_COUNT}" ]; then

32 exit 0

33 fi

37 -echo "--- Running Changelog generation ---"

38 -.travis/generate_changelog.sh "${LAST_TAG}" "${COMMITS_SINCE_RELEASE}" || echo "Changelog generation has failed, this is a soft error, process continues"

39 -

40 -echo "--- Build && publish docker images ---"

41 -# Do not fail artifacts creation if docker fails. We will be restructuring this on a follow up PR

42 -packaging/docker/build.sh && packaging/docker/publish.sh || echo "Failed to build and publish docker images"

35 +if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then

36 + echo "Beta mode -- nothing to do for ${TRAVIS_REPO_SLUG} on the nightlies script, bye"

37 + exit 0

38 +fi

44 -echo "--- Build artifacts ---"

45 -.travis/create_artifacts.sh

40 +echo "--- Running Changelog generation ---"

41 +.travis/generate_changelog_for_nightlies.sh "${LAST_TAG}" "${COMMITS_SINCE_RELEASE}" || echo "Changelog generation has failed, this is a soft error, process continues"

43 exit "${FAIL}"

.travis/releaser.sh deleted

-96

@@ -1,96 +0,0 @@

1 -#!/bin/bash

2 -# SPDX-License-Identifier: MIT

4 -# Permission to copy and modify is granted under the MIT license

5 -#

6 -# Original script is available at https://github.com/paulfantom/travis-helper/blob/master/releasing/releaser.sh

7 -#

8 -# Script to automatically do a couple of things:

9 -# - generate a new tag according to semver (https://semver.org/)

10 -# - generate CHANGELOG.md by using https://github.com/skywinder/github-changelog-generator

11 -# - create draft of GitHub releases by using https://github.com/github/hub

12 -#

13 -# Tags are generated by searching for a keyword in last commit message. Keywords are:

14 -# - [patch] or [fix] to bump patch number

15 -# - [minor], [feature] or [feat] to bump minor number

16 -# - [major] or [breaking change] to bump major number

17 -# All keywords MUST be surrounded with square braces.

18 -#

19 -# Script uses git mechanisms for locking, so it can be used in parallel builds

20 -#

21 -# Requirements:

22 -# - GITHUB_TOKEN variable set with GitHub token. Access level: repo.public_repo

23 -# - docker

24 -

25 -set -e

26 -

27 -if [ ! -f .gitignore ]; then

28 - echo "Run as ./travis/$(basename "$0") from top level directory of git repository"

29 - exit 1

30 -fi

31 -

32 -export GIT_MAIL="bot@netdata.cloud"

33 -export GIT_USER="netdatabot"

34 -echo "--- Initialize git configuration ---"

35 -git config user.email "${GIT_MAIL}"

36 -git config user.name "${GIT_USER}"

37 -git checkout master

38 -git pull

39 -

40 -echo "---- FIGURING OUT TAGS ----"

41 -# tagger.sh is sourced since we need environment variables it sets

42 -#shellcheck source=/dev/null

43 -source .travis/tagger.sh || exit 0

44 -# variable GIT_TAG is produced by tagger.sh script

45 -

46 -echo "---- UPDATE VERSION FILE ----"

47 -echo "$GIT_TAG" >packaging/version

48 -git add packaging/version

49 -

50 -echo "---- GENERATE CHANGELOG -----"

51 -./.travis/generate_changelog_for_release.sh

52 -git add CHANGELOG.md

53 -

54 -echo "---- COMMIT AND PUSH CHANGES ----"

55 -git commit -m "[ci skip] release $GIT_TAG"

56 -git tag "$GIT_TAG" -a -m "Automatic tag generation for travis build no. $TRAVIS_BUILD_NUMBER"

57 -git push "https://${GITHUB_TOKEN}:@$(git config --get remote.origin.url | sed -e 's/^https:\/\///')"

58 -git push "https://${GITHUB_TOKEN}:@$(git config --get remote.origin.url | sed -e 's/^https:\/\///')" --tags

59 -# After those operations output of command `git describe` should be identical with a value of GIT_TAG

60 -

61 -if [[ $(git describe) =~ -rc* ]]; then

62 - echo "This is a release candidate tag, exiting without artifact building"

63 - exit 0

64 -fi

65 -

66 -echo "---- CREATING TAGGED DOCKER CONTAINERS ----"

67 -./packaging/docker/build.sh

68 -./packaging/docker/publish.sh

69 -

70 -echo "---- CREATING RELEASE ARTIFACTS -----"

71 -# Artifacts are stored in `artifacts/` directory

72 -./.travis/create_artifacts.sh

73 -

74 -echo "---- CREATING RELEASE DRAFT WITH ASSETS -----"

75 -# Download hub

76 -HUB_VERSION=${HUB_VERSION:-"2.5.1"}

77 -wget "https://github.com/github/hub/releases/download/v${HUB_VERSION}/hub-linux-amd64-${HUB_VERSION}.tgz" -O "/tmp/hub-linux-amd64-${HUB_VERSION}.tgz"

78 -tar -C /tmp -xvf "/tmp/hub-linux-amd64-${HUB_VERSION}.tgz"

79 -export PATH=$PATH:"/tmp/hub-linux-amd64-${HUB_VERSION}/bin"

80 -

81 -# Create a release draft

82 -if [ -z ${GIT_TAG+x} ]; then

83 - echo "Variable GIT_TAG is not set. Something went terribly wrong! Exiting."

84 - exit 1

85 -fi

86 -if [ "${GIT_TAG}" != "$(git tag --points-at)" ]; then

87 - echo "ERROR! Current commit is not tagged. Stopping release creation."

88 - exit 1

89 -fi

90 -until hub release create --draft \

91 - -a "artifacts/netdata-${GIT_TAG}.tar.gz" \

92 - -a "artifacts/netdata-${GIT_TAG}.gz.run" \

93 - -a "artifacts/sha256sums.txt" \

94 - -m "${GIT_TAG}" "${GIT_TAG}"; do

95 - sleep 5

96 -done

.travis/run_install_with_dist_file.sh new

+48

@@ -0,0 +1,48 @@

1 +#!/usr/bin/env bash

2 +#

3 +# This script is evaluating netdata installation with the source from make dist

4 +#

5 +# Copyright: SPDX-License-Identifier: GPL-3.0-or-later

6 +#

7 +# Author : Pavlos Emm. Katsoulakis <paul@netdata.cloud)

8 +#

9 +set -e

10 +

11 +# If we are not in netdata git repo, at the top level directory, fail

12 +TOP_LEVEL=$(basename "$(git rev-parse --show-toplevel)")

13 +CWD=$(git rev-parse --show-cdup || echo "")

14 +if [ -n "${CWD}" ] || [ ! "${TOP_LEVEL}" == "netdata" ]; then

15 + echo "Run as .travis/$(basename "$0") from top level directory of netdata git repository"

16 + echo "Changelog generation process aborted"

17 + exit 1

18 +fi

19 +

20 +echo "Initiating dist archive contents validation"

21 +DIST_FILE_FROM_GIT="netdata-$(git describe).tar.gz"

22 +DIST_FILE_FROM_FILE="netdata-$(tr -d '\n' < packaging/version).tar.bgz"

23 +if [ -f ${DIST_FILE_FROM_GIT} ]; then

24 + DIST_FILE="${DIST_FILE_FROM_GIT}"

25 +elif [ -f ${DIST_FILE_FROM_FILE} ]; then

26 + DIST_FILE="${DIST_FILE_FROM_FILE}"

27 +else

28 + echo "I could not find netdata distfile. Nor ${DIST_FILE_FROM_GIT} or ${DIST_FILE_FROM_FILE} exist"

29 + exit 1

30 +fi

31 +

32 +echo "Opening dist archive ${DIST_FILE}"

33 +tar -xovf "${DIST_FILE}"

34 +NETDATA_DIST_FOLDER=$(echo ${DIST_FILE} | cut -d. -f1,2,3)

35 +if [ ! -d ${NETDATA_DIST_FOLDER} ]; then

36 + echo "I could not locate folder ${NETDATA_DIST_FOLDER}, something went wrong failing the test"

37 + exit 1

38 +fi

39 +

40 +echo "Entering ${NETDATA_DIST_FOLDER} and starting docker compilation"

41 +cd ${NETDATA_DIST_FOLDER}

42 +docker run -it -v "${PWD}:/code:rw" -w /code "netdata/os-test:centos7" /bin/bash -c "./netdata-installer.sh --dont-wait --install /tmp && echo \"Validating netdata instance is running\" && wget -O'-' 'http://127.0.0.1:19999/api/v1/info' | grep version"

43 +

44 +echo "Installation completed with no errors! Removing temporary folders"

45 +

46 +# TODO: Travis give me a permission denied on some files here, i made it a soft error until i figure out what is wrong

47 +cd -

48 +rm -rf ${NETDATA_DIST_FOLDER} || echo "I could not remove temporary directory, make sure you delete ${NETDATA_DIST_FOLDER} by yourself if this wasn't run over ephemeral storage"

.travis/tagger.sh

+30 -12

@@ -1,7 +1,4 @@

1 #!/bin/bash

2 -# SPDX-License-Identifier: MIT

4 -# Permission to copy and modify is granted under the MIT license

2 #

3 # Original script is available at https://github.com/paulfantom/travis-helper/blob/master/releasing/releaser.sh

4 #

@@ -14,20 +11,35 @@

11 # Requirements:

12 # - GITHUB_TOKEN variable set with GitHub token. Access level: repo.public_repo

13 # - git-semver python package (pip install git-semver)

17 -

18 -# exported variables are needed by releaser.sh

14 +#

15 +# Note: Exported variables needed by .travis/draft_release.sh

16 +#

17 +# Original script is available at https://github.com/paulfantom/travis-helper/blob/master/releasing/releaser.sh

18 +#

19 +# Copyright: SPDX-License-Identifier: GPL-3.0-or-later

20 +#

21 +# Author : Pawel Krupa (paulfantom)

22 +# Author : Pavlos Emm. Katsoulakis (paul@netdata.cloud)

24 set -e

22 -if [ ! -f .gitignore ]; then

23 - echo "Run as ./travis/$(basename "$0") from top level directory of git repository"

24 - exit 1

26 +

27 +# If we are not in netdata git repo, at the top level directory, fail

28 +TOP_LEVEL=$(basename "$(git rev-parse --show-toplevel)")

29 +CWD=$(git rev-parse --show-cdup || echo "")

30 +if [ -n "${CWD}" ] || [ ! "${TOP_LEVEL}" == "netdata" ]; then

31 + echo "Run as .travis/$(basename "$0") from top level directory of netdata git repository"

32 + echo "Changelog generation process aborted"

33 + exit 1

34 fi

36 +

37 # Figure out what will be new release candidate tag based only on previous ones.

38 # This assumes that RELEASES are in format of "v0.1.2" and prereleases (RCs) are using "v0.1.2-rc0"

29 -function release_candidate() {

39 +function set_tag_release_candidate() {

40 LAST_TAG=$(git semver)

41 + echo "Last tag found is: ${LAST_TAG}"

42 +

43 if [[ $LAST_TAG =~ -rc* ]]; then

44 VERSION=$(echo "$LAST_TAG" | cut -d'-' -f 1)

45 LAST_RC=$(echo "$LAST_TAG" | cut -d'c' -f 2)

@@ -35,25 +47,31 @@ function release_candidate() {

47 else

48 VERSION="$(git semver --next-minor)"

49 RC=0

50 + echo "Warning: Will set version to ${VERSION} (Last tag: ${LAST_TAG}) while tagged for release candidate generation"

51 fi

39 - GIT_TAG="v$VERSION-rc$RC"

52 + GIT_TAG="v${VERSION}-rc${RC}"

53 }

55 +echo "Determining TAG"

56 +

57 # Check if current commit is tagged or not

58 GIT_TAG=$(git tag --points-at)

59 +

60 if [ -z "${GIT_TAG}" ]; then

61 git semver

62 # Figure out next tag based on commit message

47 - echo "Last commit message: $TRAVIS_COMMIT_MESSAGE"

63 + echo "Last commit message: ${TRAVIS_COMMIT_MESSAGE}"

64 case "${TRAVIS_COMMIT_MESSAGE}" in

65 *"[netdata patch release]"*) GIT_TAG="v$(git semver --next-patch)" ;;

66 *"[netdata minor release]"*) GIT_TAG="v$(git semver --next-minor)" ;;

67 *"[netdata major release]"*) GIT_TAG="v$(git semver --next-major)" ;;

52 - *"[netdata release candidate]"*) release_candidate ;;

68 + *"[netdata release candidate]"*) set_tag_release_candidate ;;

69 *)

70 echo "Keyword not detected. Exiting..."

71 exit 0

72 ;;

73 esac

74 fi

75 +

76 +echo "Setting up GIT_TAG to ${GIT_TAG}"

77 export GIT_TAG

tests/installer/slack.sh

+41 -3

@@ -2,7 +2,7 @@

2 # BASH Lib: Simple incoming webhook for slack integration.

3 #

4 # The script expects the following parameters to be defined by the upper layer:

5 -# SLACK_INCOMING_WEBHOOK_URL

5 +# SLACK_NOTIFY_WEBHOOK_URL

6 # SLACK_BOT_NAME

7 # SLACK_CHANNEL

8 #

@@ -11,6 +11,44 @@

11 # Author: Pavlos Emm. Katsoulakis <paul@netdata.cloud

13 post_message() {

14 - MESSAGE="$1"

15 - curl -X POST --data-urlencode "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_BOT_NAME}\", \"text\": \"${MESSAGE}\", \"icon_emoji\": \":space_invader:\"}" ${SLACK_INCOMING_WEBHOOK_URL}

14 + TYPE="$1"

15 + MESSAGE="$2"

16 +

17 + case "$TYPE" in

18 + "PLAIN_MESSAGE")

19 + curl -X POST --data-urlencode "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_BOT_NAME}\", \"text\": \"${MESSAGE}\", \"icon_emoji\": \":space_invader:\"}" ${SLACK_NOTIFY_WEBHOOK_URL}

20 + ;;

21 + "TRAVIS_MESSAGE")

22 + EVENT_LINE="${TRAVIS_JOB_NUMBER}: Event type '${TRAVIS_EVENT_TYPE}', on '${TRAVIS_OS_NAME}'"

23 + if [ "$TRAVIS_EVENT_TYPE}" == "pull_request" ]; then

24 + EVENT_LINE="${TRAVIS_JOB_NUMBER}: Event type '${TRAVIS_EVENT_TYPE}' #${TRAVIS_PULL_REQUEST}, on '${TRAVIS_OS_NAME}' "

25 + fi

26 +

27 + POST_MESSAGE="{

28 + \"text\": \"${TRAVIS_REPO_SLUG}, ${MESSAGE}\",

29 + \"attachments\": [{

30 + \"text\": \"${TRAVIS_JOB_NUMBER}: Event type '${TRAVIS_EVENT_TYPE}', on '${TRAVIS_OS_NAME}' \",

31 + \"fallback\": \"I could not determine the build\",

32 + \"callback_id\": \"\",

33 + \"color\": \"#3AA3E3\",

34 + \"attachment_type\": \"default\",

35 + \"actions\": [

36 + {

37 + \"name\": \"${TRAVIS_BUILD_NUMBER}\",

38 + \"text\": \"Build #${TRAVIS_BUILD_NUMBER}\",

39 + \"type\": \"button\",

40 + \"url\": \"${TRAVIS_BUILD_WEB_URL}\"

41 + },

42 + {

43 + \"name\": \"${TRAVIS_JOB_NUMBER}\",

44 + \"text\": \"Job #${TRAVIS_JOB_NUMBER}\",

45 + \"type\": \"button\",

46 + \"url\": \"${TRAVIS_JOB_WEB_URL}\"

47 + }]

48 + }]

49 + }"

50 + echo "Sending ${POST_MESSAGE}"

51 + curl -X POST --data-urlencode "payload=${POST_MESSAGE}" "${SLACK_NOTIFY_WEBHOOK_URL}"

52 + ;;

53 + esac

54 }