@cryptotaxi247 / netdata-1 / commits / 23973d7f2

Skip unit testing during CI when it's not needed. (#7559)

* Skip unit testing during CI when it's not needed. Our current unit testing takes almost 20 minutes on average during the CI process, which is almost twice as long as any other step, and it's only going to get longer as we get better test coverage. This commit slightly alters how we handle unit testing during CI so that only those unit tests which are actually needed get run. This is achieved by using the `$TRAVIS_COMMIT_RANGE` environment variable provided by Travis to determine which commits we're actually testing, and then using that information to query Git for what files actually changed. As of right now, the only unit testing we're doing is in the dbengine code, so we check to see if any of those files or the C unit testing files are modified by the PR and only runs the unit tests for those if they were modified. The checks are a bit more wide-ranging than they probably need to be so that we make sure to avoid any false negatives. This should speed up CI checks for PR's (as well as not monopolizing the master branch after every PR that gets merged). * Restructure to better handle PR's. The previous code did not work correctly with PR's because it ignored the fact that Travis does not appear to report the full list of commits for a PR in `TRAVIS_COMMIT_RANGE`, instead only reporting the most recent set of commits that were pushed. * Add listing of commits being checked. This will allow for verification of what is being tested, making it easier to spot issues with the detection logic. * Don't assume PR's are targeting the master branch.

Austin S. Hemmelgarn committed Dec 19, 2019 at 08:41 UTC 23973d7f2abb0649109a0099d322592167ece06d
3 files changed +90 -3
.travis.yml
+4 -2
@@ -154,9 +154,11 @@ jobs:
154
155 name: Unit Testing
156 script:
157 - - fakeroot ./netdata-installer.sh --install $HOME --dont-wait --dont-start-it --enable-plugin-nfacct --enable-plugin-freeipmi --disable-lto
158 - - $HOME/netdata/usr/sbin/netdata -W unittest
157 + - .travis/run-unit-tests.sh
158 env: CFLAGS='-O1 -DNETDATA_INTERNAL_CHECKS=1 -DNETDATA_VERIFY_LOCKS=1'
159 + # We need a full deep clone for the checks done by the unit testing script to work.
160 + git:
161 + depth: false
162 after_failure: post_message "TRAVIS_MESSAGE" "Unit testing failed"
163
164 - name: Build/Install for ubuntu 18.04 (not containerized)
.travis/README.md
+1 -1
@@ -63,7 +63,7 @@ that our product meets certain epxectations. At the current stage, we are focusi
63 like installing in different distributions, running the full lifecycle of install-run-update-install and so on.
64 We are still working on enriching this with more and more use cases, to get us closer to achieving full stability of our software.
65 Briefly we currently evaluate the following activities:
66 -- Basic software unit testing
66 +- Basic software unit testing (only run when changes happen that require it)
67 - Non containerized build and install on ubuntu 14.04
68 - Non containerized build and install on ubuntu 18.04
69 - Running the full Netdata lifecycle (install, update, uninstall) on ubuntu 18.04
.travis/run-unit-tests.sh new
+85
@@ -0,0 +1,85 @@
1 +#!/usr/bin/env bash
2 +#
3 +# Unit-testing script
4 +#
5 +# This script does the following:
6 +# 1. Check whether any files were modified that would necessitate unit testing (using the `TRAVIS_COMMIT_RANGE` environment variable).
7 +# 2. If there are no changed files that require unit testing, exit successfully.
8 +# 3. Otherwise, run all the unit tests.
9 +#
10 +# We do things this way because our unit testing takes a rather long
11 +# time (average 18-19 minutes as of the original creation of this script),
12 +# so skipping it when we don't actually need it can significantly speed
13 +# up the CI process.
14 +#
15 +# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
16 +#
17 +# Author: Austin S. Hemmelgarn <austin@netdata.cloud>
18 +#
19 +# shellcheck disable=SC2230
20 +
21 +install_netdata() {
22 + echo "Installing Netdata"
23 + fakeroot ./netdata-installer.sh --install $HOME --dont-wait --dont-start-it --enable-plugin-nfacct --enable-plugin-freeipmi --disable-lto
24 +}
25 +
26 +c_unit_tests() {
27 + echo "Running C code unit tests"
28 + $HOME/netdata/usr/sbin/netdata -W unittest
29 +}
30 +
31 +run_c_unit_tests=
32 +
33 +if [ -z ${TRAVIS_COMMIT_RANGE} ] ; then
34 + # Travis gave us no commit range, so just run all the unit tests.
35 + # Per the docs, this is the case when a new branch is pushed for the first time.
36 + echo "No commit range supplied, assuming the worst case and running all unit tests."
37 + run_c_unit_tests=1
38 +else
39 + changed_paths=
40 +
41 + if [ "${TRAVIS_PULL_REQUEST}" = "false" ] ; then
42 + # This is not a PR build.
43 + COMMIT1="$(echo ${TRAVIS_COMMIT_RANGE} | cut -f 1 -d '.')"
44 + COMMIT2="$(echo ${TRAVIS_COMMIT_RANGE} | cut -f 4 -d '.')"
45 +
46 + if [ "$(git cat-file -t ${COMMIT1} 2>/dev/null)" = commit -a "$(git cat-file -t ${COMMIT2} 2>/dev/null)" = commit ] ; then
47 + # Examine the exact set of commits passed by Travis.
48 + echo "Checking commits:"
49 + git log --format=oneline --abbrev-commit ${COMMIT1}..${COMMIT2}
50 + changed_paths="$(git diff --name-only ${COMMIT1}..${COMMIT2} --)"
51 + else
52 + # We couldn't find at least one of the changesets, so this build
53 + # was probably triggered by a history rewrite. Since we can't
54 + # figure out what chnaged, we need to just run all the tests anyway.
55 + echo "Cannot determine which commits we are testing, running all unit tests."
56 + run_c_unit_tests=1
57 + fi
58 + else
59 + # This is a PR build, look at all commits from the target branch
60 + # to HEAD.
61 + echo "Checking commits:"
62 + git log --format=oneline --abbrev-commit ${TRAVIS_BRANCH}..HEAD
63 + changed_paths="$(git diff --name-only ${TRAVIS_BRANCH}..HEAD --)"
64 + fi
65 +
66 + if [ -n "${changed_paths}" ] ; then
67 + # Check for changes that would require the C code to be re-tested
68 + if (echo ${changed_paths} | grep -qE "daemon/unit_test|database") ; then
69 + echo "Commits appear to change C code with unit tests, queueing C unit tests."
70 + run_c_unit_tests=1
71 + fi
72 + fi
73 +fi
74 +
75 +if [ -z ${run_c_unit_tests} ] ; then
76 + # No tests to run, log this and exit with success
77 + echo "Commit range ${TRAVIS_COMMIT_RANGE} appears to make no changes that require unit tests, skipping unit testing."
78 + exit 0
79 +else
80 + install_netdata || exit 1
81 +
82 + if [ -n ${run_c_unit_tests} ] ; then
83 + c_unit_tests || exit 1
84 + fi
85 +fi