netdata/packaging: Introducing automatic binary packages generation and delivery for RPM types (Phase 1) (#6223)
* netdata/packaging: Introducing automatic binary packages generation and delivery for RPM types (Phase 1) With this commit we introduce our packaging toolkits and workflows to automate the delivery for RPM packages to packagecloud.io, our packaging content delivery infrastructure. At this stage we have only prepared the required flows to publish our latest stable release. Missing items for now: 1) Packaging versioning - we won't be providing increasing version numbers for the same upstream netdata version 2) Nightlies - our nightlies won't be provided by this channel, yet With this changeset, we also introduce the basic artifacts for DEB packaging generation, with the same misses mentioned above. Debian is likely not to be functional within this sprint run though. Here's a more detailed list of the changes, as they were generated from the original branch: [33m89baa00b[m netdata/packging: Prior to the PR, place the master bracnh as the place to execute packaging [33macfd2804[m export variables at before_script [33mb95b112b[m remove that README.md, we will cover details from the distributions document in packaging folder [33me52ab82e[m Update README.md [33m8df1f8e9[m [ci skip] There is no need for package all option. There is absolutely no uniformity on available images or distro per arch to do our job for all at the same time, so remove it for now [33m6fc1600c[m remove a few of the unavailable architectures from distros [33m7e2741de[m missed arch, we actually have i686 [33m64c72b4e[m Add arm64 architecture as a build option (to be refined on follow up commit for the distros not having arm64 or amd64 not to trigger build) [33mead672b4[m silence shellchecking, also fix a shameful recommendation from shellcheck so that array parsing can work [33m986d0e00[m you disappoint me shellcheck, shame on you [33ma5c11a75[m remove debug ls command [33m4b5fade6[m nit - too obvious to spot [33m7edb9215[m nits and fixes [33m06cf10b9[m fix the obvious miss - you need to the full path appended before the container name, otherwise no folder.. [33md777f8ae[m Give me a break travis, whats wrong with you -- define packages directory at the yaml config level [33mf9756642[m nits [33mee253493[m missed some old parts [33m1506105c[m we dont need container root variable here, we just need to go through the base LXC directory [33m03dc231c[m restructure packaging preparation process [33m529cdedb[m simplify, unify logic around build arch handling between creation and building [33m17624c14[m pull packages directory variable in the code. We calculate base on arch [33mc47ea326[m remove redudant package type var, also hard exit when not found the right stuff [33m8890041d[m Pull more arch dependent variables out of yaml config and into the code [33m832fbf71[m remove unused variable [33mb54d6ccb[m Move check lower [33m0b1a3a7e[m handle build arch requirement, plus some shellcheck fixes [33m07bb01fe[m missed this, should pass container name instead [33m1ab8a3a4[m we dont use build arch within this scope, ditch the check [33m6d3df1b3[m Add i386 packaging keywords [33m1b403680[m Remove arch from the descriptions and the environment variables We wont be building sections per arch in travis, to minimize config file size. for each distro we will always build both ARCHS unless the commit message instructs us to build specific architecture [33m8f38caa5[m Drop support of 42.3 [33m8e394906[m its word counting here, focus [33md3f6009c[m fix yanking command [33m5b4d7b9a[m dont call gpg flag on all distros, focus [33m7e0f4d64[m better listing when yanking packages [33mbdb2d3aa[m skipping gpg checks, for now [33m75841a67[m use newer distro instead [33ma439a9f9[m downgrade base dist [33meba0fba4[m Merge branch 'master' into package-cloud-deployer [33m241c0e10[m nits and fixeS [33maad8d8bf[m fix naming, add update process in debian preparation [33m872b15af[m Separately list rpm and srpm contents for yanking [33mb3ef3a6f[m el/8 and fedora/31 not ready yet [33md468cad7[m Null commit [33m22135a2f[m fix stages for packaging [33m1464cd6b[m Null commit [33m5d44d4f9[m implement yanking - when same version is uploaded, remove to re-add [33m0a4e2e46[m sync the repo first of all [33m9f71583c[m Generalize repo tool definition on yaml config, so that we adjust per distro as needed [33m3988eb74[m adjustments on the distro list, also remove coreutils -- not important [33m51571968[m check if its just permission issue [33m38fa8b2b[m adjust syntax [33me08ebd2b[m Due to the peculiarity of the way we do things, script out the packages preparation so that we can sudo them [33m0a3ce2c1[m Now that we solved all other issues, make sure you add rpm building dependencies to LXC [33maa0b370e[m a couple of nits and fixes 1) we need sudo within the container 2) just use wget everywhere, dont mix up too many packages for the same thing 3) do not try to add packages again, our dependency script should take care that. Single point of responsibility [33ma3f42e8d[m You need to request home directory to be created (need to review under the different distros for the right syntax though) [33md7a3d0d0[m dont go and add manually the required packages, call our dependency scriptlet [33m5c0888b7[m skip GPG checking for now [33m9aa64bbe[m dont create from the python lib [33m19f8cc6b[m Add more keywords for better grouping [33m0d5b89d7[m nits [33me260a57d[m python libs not in good shape, try running create from command line instead [33m7fa58921[m netdata/packaging/ci: Bingo - add container name on the RPM jobs [33mb99ff89e[m some nits and fixes, lets see [33m636057b2[m bring back sudo, move to /var/lib/lxc, pass -E to sudo to preserve vars [33m1e487517[m i wonder.. [33m97583506[m Update SuSE list [33m2e92649b[m fix doc - wrong distro details [33m6f53a4c2[m Initial distro document -- WIP [33md2ef5e4f[m Add some README stuff, also introduce yanking step in pre-deploy (empty script at this point) [33m465f293f[m Change the token [33m36431ab5[m Add a package cloud wrapper (required for yanking RPMs) [33m2142a024[m Auto-detect username for the beta deployment, so that others can use it. Secret key will remain as is for now, will do revisit this also, so that others can beta test the implementation too. Also include provide for the production deployment, when running on master [33mab5a85ad[m Add more special conditions, also fix spacing and style a little bit more: [33mce893eae[m build only on special conditions (branch name for now) [33m82340536[m Attempt to find how to ref template in stages [33m5375cd93[m one more thing - use ubuntu instead of download template, lets see [33m4ff26df4[m Attempt to fetch lxc-templates (not installed by default [33m52715ba2[m Bring back original implementation - tests gave some light to the problem [33m4607e7b8[m lets see, try different template [33m4a453425[m add some more debug stuff - will remove them once i figure it out [33m5ee2e03d[m Try to use the binaries for the creation, see what changes [33m95a22ef7[m another re-arrangement [33m82855f91[m Attempt to install a different version of libraries, as per another issue [33mfbda0d3a[m Attempt to debug weird container failure [33md828de8c[m debugging [33m07bad55d[m reduce the noise, when not possible to create/modify the required directories dont do anything [33m6a2bd46b[m rename [33mbba22329[m netdata/health: shellchecking - SC2236 [33m103a7df9[m Adjust naming [33mea70d482[m Run on Xenial [33mc2b3ad77[m You should sudo create the folder [33ma2f00c26[m Do not stop the build if one of the RPM fails, we want all of them to attempt to build [33mc367ab6c[m Cleanup, bring also RPM template in and lets see what happens [33mba7b441b[m more nits -- add the right replacements [33m2dfbb3f5[m move templated parts outside [33m3211bc11[m another approach, missed something [33m2a47dd17[m A more educated attempt based on other resources [33m670c35b7[m first time in anchors, most likely is wrong but got to see what travis will do [33m6c00628e[m Binary release flows and other side fixes [33m1d89e519[m Just in case, dont clean up if the env is messed up [33m06b8de6f[m start building up deb structure and flow, do some more re-arrangements also [33m05164ba6[m longshot - bring the distro vars on the top to attempt to build a matrix. Also a fix: create the package folder prior populating it [33mc9334c24[m Move RPM code to separate folder, make room for the deb implementations [33m8474c823[m do not copy over everything, only the RPM directories [33m13d8ac0a[m netdata/packages: attempt to copy over the folder to a different location [33m3cb0edf6[m remove temporarily to check deploy process [33m79d3b69c[m Permissions [33m05c3d4bd[m Add listing of container fs contents to see whats the result [33meeee2249[m fix container path, missed to adjust this after changing container name format [33m90a70b9c[m Add dependencies [33m34c3e1fa[m Dont forget to create the rpmbuild structure required [33m9bfb6026[m reverse the order, so that we first add all required packages [33m663af0e5[m Attempt to add all as user rather than w00t [33m0fa68530[m more fixes 1) install wget within the container, so that we can then fetch our source 2) factor out the command execution, for cleaner code [33m23cc4517[m better logging [33mda58043b[m skip formatting, do the traditional way for now [33m818508c7[m more fixes 1) escape underscore too. Actually previously the complain was about underscore, i wasnt paying attention 2) Parse all command results and make sure you break the script if something goes wrong, otherwise travis wont know the failures [33m78b9be3a[m Add more debug messages and also fix the non-escaped character % in string [33m2037f2c5[m fixes and next steps 1) Use sudo, attempt to make privileged containers this way in case there is something messed up with homedirs in travis (long shot) 2) Implement the build process for the next step (not tested, we 'll see how it works) 3) Change the container name, to something more specific to the build we are preparing to help us identify the container more accurately (and avoid possible conflicts) [33m9dac37bf[m revert attempt, obviously it didnt provide any value [33m2d975a7d[m Attempt to catch any stacktrace, if any is raised [33m93826afe[m Reinstate skip_cleanup variable, as its needed for deploy anyway [33m54fd126a[m That is weird, but seems like lxc is there on subsequent runs, make sure you clear before you start [33mf876e6a8[m adjustments 1) add version string statically for now, for testing 2) bring more vars in 3) create the container from python all the way [33m67df8903[m Flesh out the workflow for building up the builder environment on one distro [33m0906b41e[m change distro, dont forget [33m17e78ee1[m fix distro keyword [33m26ca17d7[m Create an experimental stage and bring onboard an encrypted token [33m4b0d8ce3[m Introduce an experimental package cloud deployment stage * netdata/packaging: Remove hardcoded tmp director as per Codacy feedback * netdata/packaging: Update distributions document Update DISTRIBUTIONS.md based on Codacy warnings (round 1) * netdata/packaging: lets try to attend a few errors * Removing DISTRIBUTIONS.md from this PR As this is part of a separate task, it will be added on a separate branch and linked to the respective task, to unblock RPM generation
Paul Emm. Katsoulakis committed
Jun 9, 2019 at 11:50 UTC
abdd4b0e645681f35e2d5c01fd0f60d0b700965a
13 files changed
+1003
-8
.travis.yml
+321
-4
@@ -4,11 +4,15 @@ language: c
4
services:
5
- docker
6
7
+
8
+
9
# This is a hook to help us introduce "soft" errors on our process
10
matrix:
11
allow_failures:
12
- env: ALLOW_SOFT_FAILURE_HERE=true
13
14
+
15
+
16
# Install dependencies for all, once
17
#
18
install:
@@ -18,11 +22,15 @@ install:
22
- docker info
23
- source tests/installer/slack.sh
24
25
+
26
+
27
# Setup notification system
28
#
29
notifications:
30
webhooks: https://app.fossa.io/hooks/travisci
31
32
+
33
+
34
# Define the stage sequence and conditionals
35
#
36
stages:
@@ -46,6 +54,149 @@ stages:
54
- name: Publish for release
55
if: branch = master AND type != pull_request AND type != cron AND commit_message =~ /(\[netdata release candidate\]|\[netdata major release\]|\[netdata minor release\]|\[netdata patch release\])/
56
57
+ # Build DEB packages under special conditions
58
+ # Ubuntu
59
+- name: "Package ubuntu/disco"
60
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package arm64 DEB Ubuntu\]|\[Package arm64 DEB\]|\[Package i386 DEB Ubuntu\]|\[Package i386 DEB\]|\[Package AMD64 DEB Ubuntu\]|\[Package AMD64 DEB\])/
61
+- name: "Package ubuntu/cosmic"
62
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package arm64 DEB Ubuntu\]|\[Package arm64 DEB\]|\[Package i386 DEB Ubuntu\]|\[Package i386 DEB\]|\[Package AMD64 DEB Ubuntu\]|\[Package AMD64 DEB\])/
63
+- name: "Package ubuntu/bionic"
64
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package arm64 DEB Ubuntu\]|\[Package arm64 DEB\]|\[Package i386 DEB Ubuntu\]|\[Package i386 DEB\]|\[Package AMD64 DEB Ubuntu\]|\[Package AMD64 DEB\])/
65
+- name: "Package ubuntu/artful"
66
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package arm64 DEB Ubuntu\]|\[Package arm64 DEB\]|\[Package i386 DEB Ubuntu\]|\[Package i386 DEB\]|\[Package AMD64 DEB Ubuntu\]|\[Package AMD64 DEB\])/
67
+
68
+ # Debian
69
+- name: "Package debian/buster"
70
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package arm64 DEB Debian\]|\[Package arm64 DEB\]|\[Package i386 DEB Debian\]|\[Package i386 DEB\]|\[Package AMD64 DEB Debian\]|\[Package AMD64 DEB\])/
71
+- name: "Package debian/stretch"
72
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package arm64 DEB Debian\]|\[Package arm64 DEB\]|\[Package i386 DEB Debian\]|\[Package i386 DEB\]|\[Package AMD64 DEB Debian\]|\[Package AMD64 DEB\])/
73
+- name: "Package debian/jessie"
74
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package arm64 DEB Debian\]|\[Package arm64 DEB\]|\[Package i386 DEB Debian\]|\[Package i386 DEB\]|\[Package AMD64 DEB Debian\]|\[Package AMD64 DEB\])/
75
+- name: "Package debian/wheezy"
76
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package arm64 DEB Debian\]|\[Package arm64 DEB\]|\[Package i386 DEB Debian\]|\[Package i386 DEB\]|\[Package AMD64 DEB Debian\]|\[Package AMD64 DEB\])/
77
+
78
+ # Build RPM packages under special conditions
79
+ # Enterprise linux (Covers CentOS, Redhat, Amazon linux)
80
+- name: "Package Enterprise Linux 7"
81
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package arm64 RPM Enterprise Linux\]|\[Package arm64 RPM\]|\[Package i386 RPM Enterprise Linux\]|\[Package i386 RPM\]|\[Package AMD64 RPM Enterprise Linux\]|\[Package AMD64 RPM\])/
82
+- name: "Package Enterprise linux 6"
83
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package i386 RPM Enterprise Linux\]|\[Package i386 RPM\]|\[Package AMD64 RPM Enterprise Linux\]|\[Package AMD64 RPM\])/
84
+
85
+ # Fedora
86
+- name: "Package Fedora 30"
87
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package arm64 RPM Fedora\]|\[Package arm64 RPM\]|\[Package AMD64 RPM Fedora\]|\[Package AMD64 RPM\])/
88
+- name: "Package Fedora 29"
89
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package arm64 RPM Fedora\]|\[Package arm64 RPM\]|\[Package AMD64 RPM Fedora\]|\[Package AMD64 RPM\])/
90
+- name: "Package Fedora 28"
91
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package arm64 RPM Fedora\]|\[Package arm64 RPM\]|\[Package AMD64 RPM Fedora\]|\[Package AMD64 RPM\])/
92
+
93
+ # OpenSuSE
94
+- name: "Package OpenSuSE 15.1"
95
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package arm64 RPM openSuSE\]|\[Package arm64 RPM\]|\[Package AMD64 RPM openSuSE\]|\[Package AMD64 RPM\])/
96
+- name: "Package OpenSuSE 15.0"
97
+ if: type != cron AND branch = master AND commit_message =~ /(\[Package arm64 RPM openSuSE\]|\[Package arm64 RPM\]|\[Package AMD64 RPM openSuSE\]|\[Package AMD64 RPM\])/
98
+
99
+
100
+
101
+# DEB and RPM template flows
102
+- stage: &_RPM_TEMPLATE
103
+ name: "Build & Publish RPM package"
104
+ before_install:
105
+ - sudo apt-get install -y wget lxc lxc-templates
106
+ - source tests/installer/slack.sh
107
+ before_script:
108
+ - post_message "TRAVIS_MESSAGE" "Starting package preparation and publishing for ${BUILD_STRING}.${BUILD_ARCH}"
109
+ - export PACKAGES_DIRECTORY="$(mktemp -d -t netdata-packaging-contents-dir-XXXXXX)" && echo "Created packaging directory ${PACKAGES_DIRECTORY}"
110
+ - export BUILD_VERSION=$(cat packaging/version | cut -d'-' -f1) # We always build on the latest release
111
+ script:
112
+ - echo "Creating LXC environment for the build" && sudo -E .travis/package_management/create_lxc_for_build.sh
113
+ - echo "Building package in container" && sudo -E .travis/package_management/build_package_in_container.sh
114
+ - sudo chmod -R 755 "/var/lib/lxc"
115
+ - echo "Preparing RPM packaging contents for upload" && sudo -E .travis/package_management/prepare_packages.sh
116
+ after_failure: post_message "TRAVIS_MESSAGE" "Failed to build RPM for ${BUILD_STRING}.${BUILD_ARCH}"
117
+ before_deploy:
118
+ - .travis/package_management/yank_stale_rpm.sh "${PACKAGES_DIRECTORY}" "${BUILD_STRING}" || echo "No stale RPM found"
119
+ deploy:
120
+ # Beta packages deployment
121
+ - provider: packagecloud
122
+ repository: "netdata"
123
+ username: "$(echo ${TRAVIS_REPO_SLUG} | cut -d'/' -f1)"
124
+ token: "${PKG_CLOUD_TOKEN}"
125
+ dist: "${BUILD_STRING}"
126
+ local_dir: "${PACKAGES_DIRECTORY}"
127
+ skip_cleanup: true
128
+ on:
129
+ # Only deploy on ${USER}/netdata, master branch, when packages directory is created
130
+ repo: ${TRAVIS_REPO_SLUG}
131
+ branch: master
132
+ condition: -d "${PACKAGES_DIRECTORY}"
133
+ # Production release packages deployment
134
+ - provider: packagecloud
135
+ repository: "netdata"
136
+ username: "netdata"
137
+ token: "${PKG_CLOUD_TOKEN}"
138
+ dist: "${BUILD_STRING}"
139
+ local_dir: "${PACKAGES_DIRECTORY}"
140
+ skip_cleanup: true
141
+ on:
142
+ # Only deploy on ${USER}/netdata, master branch, when packages directory is created
143
+ repo: "${TRAVIS_REPO_SLUG}"
144
+ branch: master
145
+ condition: -d "${PACKAGES_DIRECTORY}"
146
+ after_deploy:
147
+ - if [ -n "${BUILDER_NAME}" ]; then rm -rf /home/${BUILDER_NAME}/* && echo "Cleared /home/${BUILDER_NAME} directory" || echo "Failed to clean /home/${BUILDER_NAME} directory"; fi;
148
+ - if [ -d "${PACKAGES_DIRECTORY}" ]; then rm -rf "${PACKAGES_DIRECTORY}"; fi;
149
+
150
+
151
+
152
+- stage: &_DEB_TEMPLATE
153
+ name: "Build & Publish DEB package"
154
+ before_install:
155
+ - sudo apt-get install -y wget lxc lxc-templates
156
+ - source tests/installer/slack.sh
157
+ before_script:
158
+ - post_message "TRAVIS_MESSAGE" "Starting package preparation and publishing for ${BUILD_STRING}.${BUILD_ARCH}"
159
+ - export PACKAGES_DIRECTORY="$(mktemp -d -t netdata-packaging-contents-dir-XXXXXX)" && echo "Created packaging directory ${PACKAGES_DIRECTORY}"
160
+ - export BUILD_VERSION=$(cat packaging/version | cut -d'-' -f1) # We always build on the latest release
161
+ script:
162
+ - echo "Creating LXC environment for the build" && sudo -E .travis/package_management/create_lxc_for_build.sh
163
+ - echo "Building package in container" && sudo -E .travis/package_management/build_package_in_container.sh
164
+ - sudo chmod -R 755 "/var/lib/lxc"
165
+ - echo "Preparing DEB packaging contents for upload" && sudo -E .travis/package_management/prepare_packages.sh
166
+ after_failure: post_message "TRAVIS_MESSAGE" "Failed to build DEB for ${BUILD_STRING}.${BUILD_ARCH}"
167
+ before_deploy:
168
+ - .travis/package_management/yank_stale_rpm.sh "${PACKAGES_DIRECTORY}" "${BUILD_STRING}" || echo "No stale DEB found"
169
+ deploy:
170
+ # Beta packages deployment
171
+ - provider: packagecloud
172
+ repository: "netdata"
173
+ username: "$(echo ${TRAVIS_REPO_SLUG} | cut -d'/' -f1)"
174
+ token: "${PKG_CLOUD_TOKEN}"
175
+ dist: "${BUILD_STRING}"
176
+ local_dir: "${PACKAGES_DIRECTORY}"
177
+ skip_cleanup: true
178
+ on:
179
+ # Only deploy on ${USER}/netdata, master branch, when build-area directory is created
180
+ repo: ${TRAVIS_REPO_SLUG}
181
+ branch: master
182
+ condition: -d "${PACKAGES_DIRECTORY}"
183
+ # Production release packages deployment
184
+ - provider: packagecloud
185
+ repository: "netdata"
186
+ username: "netdata"
187
+ token: "${PKG_CLOUD_TOKEN}"
188
+ dist: "${BUILD_STRING}"
189
+ local_dir: "${PACKAGES_DIRECTORY}"
190
+ skip_cleanup: true
191
+ on:
192
+ # Only deploy on ${USER}/netdata, master branch, when build-area directory is created
193
+ repo: "${TRAVIS_REPO_SLUG}"
194
+ branch: master
195
+ condition: -d "${PACKAGES_DIRECTORY}"
196
+ after_deploy:
197
+ - if [ -n "${BUILDER_NAME}" ]; then rm -rf /home/${BUILDER_NAME}/* && echo "Cleared /home/${BUILDER_NAME} directory" || echo "Failed to clean /home/${BUILDER_NAME} directory"; fi;
198
+ - if [ -d "${PACKAGES_DIRECTORY}" ]; then rm -rf "${PACKAGES_DIRECTORY}"; fi;
199
+
200
201
202
# Define stage implementation details
@@ -181,7 +332,173 @@ jobs:
332
333
334
184
- # We only publish if a TAG has been set during packaging
335
+ # ###### Packaging workflow section ######
336
+ # References:
337
+ # https://us.images.linuxcontainers.org
338
+ # https://packagecloud.io/docs#install_repo
339
+
340
+ # Ubuntu distros build
341
+ #
342
+ - stage:
343
+ <<: *_DEB_TEMPLATE
344
+ stage: "Package ubuntu/disco"
345
+ env:
346
+ - BUILDER_NAME="builder" BUILD_DISTRO="ubuntu" BUILD_RELEASE="disco" BUILD_STRING="ubuntu/disco"
347
+ - PACKAGE_TYPE="deb" REPO_TOOL="apt-get"
348
+ - ALLOW_SOFT_FAILURE_HERE=true
349
+
350
+
351
+
352
+ - stage:
353
+ <<: *_DEB_TEMPLATE
354
+ stage: "Package ubuntu/cosmic"
355
+ env:
356
+ - BUILDER_NAME="builder" BUILD_DISTRO="ubuntu" BUILD_RELEASE="cosmic" BUILD_STRING="ubuntu/cosmic"
357
+ - PACKAGE_TYPE="deb" REPO_TOOL="apt-get"
358
+ - ALLOW_SOFT_FAILURE_HERE=true
359
+
360
+
361
+
362
+ - stage:
363
+ <<: *_DEB_TEMPLATE
364
+ stage: "Package ubuntu/bionic"
365
+ env:
366
+ - BUILDER_NAME="builder" BUILD_DISTRO="ubuntu" BUILD_RELEASE="bionic" BUILD_STRING="ubuntu/bionic"
367
+ - PACKAGE_TYPE="deb" REPO_TOOL="apt-get"
368
+ - ALLOW_SOFT_FAILURE_HERE=true
369
+
370
+
371
+
372
+ - stage:
373
+ <<: *_DEB_TEMPLATE
374
+ stage: "Package ubuntu/artful"
375
+ env:
376
+ - BUILDER_NAME="builder" BUILD_DISTRO="ubuntu" BUILD_RELEASE="artful" BUILD_STRING="ubuntu/artful"
377
+ - PACKAGE_TYPE="deb" REPO_TOOL="apt-get"
378
+ - ALLOW_SOFT_FAILURE_HERE=true
379
+
380
+
381
+
382
+ # Debian distros build
383
+ - stage:
384
+ <<: *_DEB_TEMPLATE
385
+ stage: "Package debian/buster"
386
+ env:
387
+ - BUILDER_NAME="builder" BUILD_DISTRO="debian" BUILD_RELEASE="buster" BUILD_STRING="debian/buster"
388
+ - PACKAGE_TYPE="deb" REPO_TOOL="apt-get"
389
+ - ALLOW_SOFT_FAILURE_HERE=true
390
+
391
+
392
+
393
+ - stage:
394
+ <<: *_DEB_TEMPLATE
395
+ stage: "Package debian/stretch"
396
+ env:
397
+ - BUILDER_NAME="builder" BUILD_DISTRO="debian" BUILD_RELEASE="stretch" BUILD_STRING="debian/stretch"
398
+ - PACKAGE_TYPE="deb" REPO_TOOL="apt-get"
399
+ - ALLOW_SOFT_FAILURE_HERE=true
400
+
401
+
402
+
403
+ - stage:
404
+ <<: *_DEB_TEMPLATE
405
+ stage: "Package debian/jessie"
406
+ env:
407
+ - BUILDER_NAME="builder" BUILD_DISTRO="debian" BUILD_RELEASE="jessie" BUILD_STRING="debian/jessie"
408
+ - PACKAGE_TYPE="deb" REPO_TOOL="apt-get"
409
+ - ALLOW_SOFT_FAILURE_HERE=true
410
+
411
+
412
+
413
+ - stage:
414
+ <<: *_DEB_TEMPLATE
415
+ stage: "Package debian/wheezy"
416
+ env:
417
+ - BUILDER_NAME="builder" BUILD_DISTRO="debian" BUILD_RELEASE="wheezy" BUILD_STRING="debian/wheezy"
418
+ - PACKAGE_TYPE="deb" REPO_TOOL="apt-get"
419
+ - ALLOW_SOFT_FAILURE_HERE=true
420
+
421
+
422
+
423
+ # Enterprise linux builds (Centos, Redhat, Amazon linux (el/6))
424
+ #
425
+ - stage:
426
+ <<: *_RPM_TEMPLATE
427
+ stage: "Package Enterprise Linux 7"
428
+ env:
429
+ - BUILDER_NAME="builder" BUILD_DISTRO="centos" BUILD_RELEASE="7" BUILD_STRING="el/7"
430
+ - PACKAGE_TYPE="rpm" REPO_TOOL="yum"
431
+ - ALLOW_SOFT_FAILURE_HERE=true
432
+
433
+
434
+
435
+ - stage:
436
+ <<: *_RPM_TEMPLATE
437
+ stage: "Package Enterprise linux 6"
438
+ env:
439
+ - BUILDER_NAME="builder" BUILD_DISTRO="centos" BUILD_RELEASE="6" BUILD_STRING="el/6"
440
+ - PACKAGE_TYPE="rpm" REPO_TOOL="yum"
441
+ - ALLOW_SOFT_FAILURE_HERE=true
442
+
443
+
444
+
445
+ # Fedora distros build
446
+ #
447
+ - stage:
448
+ <<: *_RPM_TEMPLATE
449
+ stage: "Package Fedora 30"
450
+ env:
451
+ - BUILDER_NAME="builder" BUILD_DISTRO="fedora" BUILD_RELEASE="30" BUILD_STRING="fedora/30"
452
+ - PACKAGE_TYPE="rpm" REPO_TOOL="dnf"
453
+ - ALLOW_SOFT_FAILURE_HERE=true
454
+
455
+
456
+
457
+ - stage:
458
+ <<: *_RPM_TEMPLATE
459
+ stage: "Package Fedora 29"
460
+ env:
461
+ - BUILDER_NAME="builder" BUILD_DISTRO="fedora" BUILD_RELEASE="29" BUILD_STRING="fedora/29"
462
+ - PACKAGE_TYPE="rpm" REPO_TOOL="dnf"
463
+ - ALLOW_SOFT_FAILURE_HERE=true
464
+
465
+
466
+
467
+ - stage:
468
+ <<: *_RPM_TEMPLATE
469
+ stage: "Package Fedora 28"
470
+ env:
471
+ - BUILDER_NAME="builder" BUILD_DISTRO="fedora" BUILD_RELEASE="28" BUILD_STRING="fedora/28"
472
+ - PACKAGE_TYPE="rpm" REPO_TOOL="dnf"
473
+ - ALLOW_SOFT_FAILURE_HERE=true
474
+
475
+
476
+
477
+ # Opensuse distros build
478
+ #
479
+ - stage:
480
+ <<: *_RPM_TEMPLATE
481
+ stage: "Package OpenSuSE 15.1"
482
+ env:
483
+ - BUILDER_NAME="builder" BUILD_DISTRO="opensuse" BUILD_RELEASE="15.0" BUILD_STRING="opensuse/15.1"
484
+ - PACKAGE_TYPE="rpm" REPO_TOOL="zypper"
485
+ - ALLOW_SOFT_FAILURE_HERE=true
486
+
487
+
488
+
489
+ - stage:
490
+ <<: *_RPM_TEMPLATE
491
+ stage: "Package OpenSuSE 15.0"
492
+ env:
493
+ - BUILDER_NAME="builder" BUILD_DISTRO="opensuse" BUILD_RELEASE="15.0" BUILD_STRING="opensuse/15.0"
494
+ - PACKAGE_TYPE="rpm" REPO_TOOL="zypper"
495
+ - ALLOW_SOFT_FAILURE_HERE=true
496
+ # ###### End of packaging workflow section ###### #
497
+ # ############################################### #
498
+
499
+
500
+
501
+ # We only publish if a TAG has been set during packaging
502
- stage: Publish for release
503
504
name: Build & Publish docker images
@@ -218,7 +535,7 @@ jobs:
535
536
537
221
- # This is the nightly pre-execution step (Jobs, preparatory steps for nightly, etc)
538
+ # This is the nightly pre-execution step (Jobs, preparatory steps for nightly, etc)
539
- stage: Nightly operations
540
541
name: Run coverity scan
@@ -242,8 +559,8 @@ jobs:
559
560
561
245
- # This is the nightly execution step
246
- #
562
+ # This is the nightly execution step
563
+ #
564
- stage: Nightly release
565
566
name: Build & Publish docker images
.travis/README.md
+46
@@ -95,3 +95,49 @@ During packaging we are preparing the release changelog information and run the
95
## Publish for release
96
The publishing stage is the most complex part in publishing. This is the stage were we generate and publish docker images,
97
prepare the release artifacts and get ready with the release draft.
98
+
99
+### Package Management workflows
100
+As part of our goal to provide the best support to our customers, we have created a set of CI workflows to automatically produce
101
+DEB and RPM for multiple distributions. These workflows are implemented under the templated stages '_DEB_TEMPLATE' and '_RPM_TEMPLATE'.
102
+We currently plan to actively support the following Operating Systems, with a plan to further expand this list following our users needs.
103
+
104
+### Operating systems supported
105
+The following distributions are supported
106
+- Debian versions
107
+ - Buster (TBD - not released yet, check [debian releases](https://www.debian.org/releases/) for details)
108
+ - Stretch
109
+ - Jessie
110
+ - Wheezy
111
+
112
+- Ubuntu versions
113
+ - Disco
114
+ - Cosmic
115
+ - Bionic
116
+ - artful
117
+
118
+- Enterprise Linux versions (Covers Redhat, CentOS, and Amazon Linux with version 6)
119
+ - Version 8 (TBD)
120
+ - Version 7
121
+ - Version 6
122
+
123
+- Fedora versions
124
+ - Version 31 (TBD)
125
+ - Version 30
126
+ - Version 29
127
+ - Version 28
128
+
129
+- OpenSuSE versions
130
+ - 15.1
131
+ - 15.0
132
+
133
+- Gentoo distributions
134
+ - TBD
135
+
136
+### Architectures supported
137
+We plan to support amd64, x86 and arm64 architectures. As of June 2019 only amd64 and x86 will become available, as we are still working on solving issues with the architecture.
138
+
139
+The Package deployment can be triggered manually by executing an empty commit with the following message pattern: `[Package PACKAGE_TYPE PACKAGE_ARCH] DESCRIBE_THE_REASONING_HERE`.
140
+Travis Yaml configuration allows the user to combine package type and architecture as necessary to regenerate the current stable release (For example tag v1.15.0 as of 4th of May 2019)
141
+Sample patterns to trigger building of packages for all AMD64 supported architecture:
142
+- '[Package AMD64 RPM]': Build & publish all amd64 available RPM packages
143
+- '[Package AMD64 DEB]': Build & publish all amd64 available DEB packages
.travis/package_management/build_package_in_container.sh
new
+82
@@ -0,0 +1,82 @@
1
+#!/usr/bin/env bash
2
+#
3
+# Entry point for package build process
4
+#
5
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
6
+#
7
+# Author : Pavlos Emm. Katsoulakis (paul@netdata.cloud)
8
+#shellcheck disable=SC1091
9
+set -e
10
+
11
+# If we are not in netdata git repo, at the top level directory, fail
12
+TOP_LEVEL=$(basename "$(git rev-parse --show-toplevel)")
13
+CWD=$(git rev-parse --show-cdup)
14
+if [ -n "$CWD" ] || [ ! "${TOP_LEVEL}" == "netdata" ]; then
15
+ echo "Run as .travis/package_management/$(basename "$0") from top level directory of netdata git repository"
16
+ echo "Docker build process aborted"
17
+ exit 1
18
+fi
19
+
20
+source .travis/package_management/functions.sh || (echo "Failed to load packaging library" && exit 1)
21
+
22
+# Check for presence of mandatory environment variables
23
+if [ -z "${BUILD_STRING}" ]; then
24
+ echo "No Distribution was defined. Make sure BUILD_STRING is set on the environment before running this script"
25
+ exit 1
26
+fi
27
+
28
+if [ -z "${BUILDER_NAME}" ]; then
29
+ echo "No builder account and container name defined. Make sure BUILDER_NAME is set on the environment before running this script"
30
+ exit 1
31
+fi
32
+
33
+if [ -z "${BUILD_DISTRO}" ]; then
34
+ echo "No build distro information defined. Make sure BUILD_DISTRO is set on the environment before running this script"
35
+ exit 1
36
+fi
37
+
38
+if [ -z "${BUILD_RELEASE}" ]; then
39
+ echo "No build release information defined. Make sure BUILD_RELEASE is set on the environment before running this script"
40
+ exit 1
41
+fi
42
+
43
+if [ -z "${PACKAGE_TYPE}" ]; then
44
+ echo "No build release information defined. Make sure PACKAGE_TYPE is set on the environment before running this script"
45
+ exit 1
46
+fi
47
+
48
+# Detect architecture and load extra variables needed
49
+detect_arch_from_commit
50
+
51
+case "${BUILD_ARCH}" in
52
+"all")
53
+ echo "* * * Building all architectures, amd64 and i386 * * *"
54
+ echo "Building for amd64.."
55
+ export CONTAINER_NAME="${BUILDER_NAME}-${BUILD_DISTRO}${BUILD_RELEASE}-amd64"
56
+ export LXC_CONTAINER_ROOT="/var/lib/lxc/${CONTAINER_NAME}/rootfs"
57
+ .travis/package_management/"${PACKAGE_TYPE}"/trigger_lxc_build.py "${CONTAINER_NAME}"
58
+
59
+ echo "Building for arm64.."
60
+ export CONTAINER_NAME="${BUILDER_NAME}-${BUILD_DISTRO}${BUILD_RELEASE}-arm64"
61
+ export LXC_CONTAINER_ROOT="/var/lib/lxc/${CONTAINER_NAME}/rootfs"
62
+ .travis/package_management/"${PACKAGE_TYPE}"/trigger_lxc_build.py "${CONTAINER_NAME}"
63
+
64
+ echo "Building for i386.."
65
+ export CONTAINER_NAME="${BUILDER_NAME}-${BUILD_DISTRO}${BUILD_RELEASE}-i386"
66
+ export LXC_CONTAINER_ROOT="/var/lib/lxc/${CONTAINER_NAME}/rootfs"
67
+ .travis/package_management/"${PACKAGE_TYPE}"/trigger_lxc_build.py "${CONTAINER_NAME}"
68
+
69
+ ;;
70
+"amd64"|"arm64"|"i386")
71
+ echo "Building for ${BUILD_ARCH}.."
72
+ export CONTAINER_NAME="${BUILDER_NAME}-${BUILD_DISTRO}${BUILD_RELEASE}-${BUILD_ARCH}"
73
+ export LXC_CONTAINER_ROOT="/var/lib/lxc/${CONTAINER_NAME}/rootfs"
74
+ .travis/package_management/"${PACKAGE_TYPE}"/trigger_lxc_build.py "${CONTAINER_NAME}"
75
+ ;;
76
+*)
77
+ echo "Unknown build architecture '${BUILD_ARCH}', nothing to do for build"
78
+ exit 1
79
+ ;;
80
+esac
81
+
82
+echo "Build process completed!"
.travis/package_management/create_lxc_for_build.sh
new
+101
@@ -0,0 +1,101 @@
1
+#!/usr/bin/env bash
2
+#
3
+# This script generates an LXC container and starts it up
4
+# Once the script completes successfully, a container has become available for usage
5
+# The container image to be used and the container name to be set, are part of variables
6
+# that must be present for the script to work
7
+#
8
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
9
+#
10
+# Author : Pavlos Emm. Katsoulakis (paul@netdata.cloud)
11
+# shellcheck disable=SC1091
12
+set -e
13
+
14
+source .travis/package_management/functions.sh || (echo "Failed to load packaging library" && exit 1)
15
+
16
+# If we are not in netdata git repo, at the top level directory, fail
17
+TOP_LEVEL=$(basename "$(git rev-parse --show-toplevel)")
18
+CWD=$(git rev-parse --show-cdup)
19
+if [ -n "$CWD" ] || [ ! "${TOP_LEVEL}" == "netdata" ]; then
20
+ echo "Run as .travis/package_management/$(basename "$0") from top level directory of netdata git repository"
21
+ echo "LXC Container creation aborted"
22
+ exit 1
23
+fi
24
+
25
+# Check for presence of mandatory environment variables
26
+if [ -z "${BUILD_STRING}" ]; then
27
+ echo "No Distribution was defined. Make sure BUILD_STRING is set on the environment before running this script"
28
+ exit 1
29
+fi
30
+
31
+if [ -z "${BUILDER_NAME}" ]; then
32
+ echo "No builder account and container name defined. Make sure BUILDER_NAME is set on the environment before running this script"
33
+ exit 1
34
+fi
35
+
36
+if [ -z "${BUILD_DISTRO}" ]; then
37
+ echo "No build distro information defined. Make sure BUILD_DISTRO is set on the environment before running this script"
38
+ exit 1
39
+fi
40
+
41
+if [ -z "${BUILD_RELEASE}" ]; then
42
+ echo "No build release information defined. Make sure BUILD_RELEASE is set on the environment before running this script"
43
+ exit 1
44
+fi
45
+
46
+if [ -z "${PACKAGE_TYPE}" ]; then
47
+ echo "No build release information defined. Make sure PACKAGE_TYPE is set on the environment before running this script"
48
+ exit 1
49
+fi
50
+
51
+# Detect architecture and load extra variables needed
52
+detect_arch_from_commit
53
+
54
+echo "Creating LXC container ${BUILDER_NAME}/${BUILD_STRING}/${BUILD_ARCH}...."
55
+
56
+case "${BUILD_ARCH}" in
57
+"all")
58
+ # i386
59
+ echo "Creating LXC Container for i386.."
60
+ export CONTAINER_NAME="${BUILDER_NAME}-${BUILD_DISTRO}${BUILD_RELEASE}-i386"
61
+ export LXC_CONTAINER_ROOT="/var/lib/lxc/${CONTAINER_NAME}/rootfs"
62
+ lxc-create -n "${CONTAINER_NAME}" -t "download" -- --dist "${BUILD_DISTRO}" --release "${BUILD_RELEASE}" --arch "i386" --no-validate
63
+
64
+ echo "Container(s) ready. Configuring container(s).."
65
+ .travis/package_management/"${PACKAGE_TYPE}"/configure_lxc_environment.py "${CONTAINER_NAME}"
66
+
67
+ # amd64
68
+ echo "Creating LXC Container for amd64.."
69
+ export CONTAINER_NAME="${BUILDER_NAME}-${BUILD_DISTRO}${BUILD_RELEASE}-amd64"
70
+ export LXC_CONTAINER_ROOT="/var/lib/lxc/${CONTAINER_NAME}/rootfs"
71
+ lxc-create -n "${CONTAINER_NAME}" -t "download" -- --dist "${BUILD_DISTRO}" --release "${BUILD_RELEASE}" --arch "amd64" --no-validate
72
+
73
+ echo "Container(s) ready. Configuring container(s).."
74
+ .travis/package_management/"${PACKAGE_TYPE}"/configure_lxc_environment.py "${CONTAINER_NAME}"
75
+
76
+ # arm64
77
+ echo "Creating LXC Container for arm64.."
78
+ export CONTAINER_NAME="${BUILDER_NAME}-${BUILD_DISTRO}${BUILD_RELEASE}-arm64"
79
+ export LXC_CONTAINER_ROOT="/var/lib/lxc/${CONTAINER_NAME}/rootfs"
80
+ lxc-create -n "${CONTAINER_NAME}" -t "download" -- --dist "${BUILD_DISTRO}" --release "${BUILD_RELEASE}" --arch "arm64" --no-validate
81
+
82
+ echo "Container(s) ready. Configuring container(s).."
83
+ .travis/package_management/"${PACKAGE_TYPE}"/configure_lxc_environment.py "${CONTAINER_NAME}"
84
+ ;;
85
+"i386"|"amd64"|"arm64")
86
+ # AMD64 or i386
87
+ echo "Creating LXC Container for ${BUILD_ARCH}.."
88
+ export CONTAINER_NAME="${BUILDER_NAME}-${BUILD_DISTRO}${BUILD_RELEASE}-${BUILD_ARCH}"
89
+ export LXC_CONTAINER_ROOT="/var/lib/lxc/${CONTAINER_NAME}/rootfs"
90
+ lxc-create -n "${CONTAINER_NAME}" -t "download" -- --dist "${BUILD_DISTRO}" --release "${BUILD_RELEASE}" --arch "${BUILD_ARCH}" --no-validate
91
+
92
+ echo "Container(s) ready. Configuring container(s).."
93
+ .travis/package_management/"${PACKAGE_TYPE}"/configure_lxc_environment.py "${CONTAINER_NAME}"
94
+ ;;
95
+*)
96
+ echo "Unknown BUILD_ARCH value '${BUILD_ARCH}' given, process failed"
97
+ exit 1
98
+ ;;
99
+esac
100
+
101
+echo "..LXC creation complete!"
.travis/package_management/deb/configure_lxc_environment.py
new
+71
@@ -0,0 +1,71 @@
1
+#!/usr/bin/env python3
2
+#
3
+# Prepare the build environment within the container
4
+# The script attaches to the running container and does the following:
5
+# 1) Create the container
6
+# 2) Start the container up
7
+# 3) Create the builder user
8
+# 4) Prepare the environment for DEB build
9
+#
10
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
11
+#
12
+# Author : Pavlos Emm. Katsoulakis <paul@netdata.cloud>
13
+
14
+import os
15
+import sys
16
+import lxc
17
+
18
+def run_command(command):
19
+ print ("Running command: %s" % command)
20
+ command_result = container.attach_wait(lxc.attach_run_command, command)
21
+
22
+ if command_result != 0:
23
+ raise Exception("Command failed with exit code %d" % command_result)
24
+
25
+if len(sys.argv) != 2:
26
+ print ('You need to provide a container name to get things started')
27
+ sys.exit(1)
28
+container_name=sys.argv[1]
29
+
30
+# Setup the container object
31
+print ("Defining container %s" % container_name)
32
+container = lxc.Container(container_name)
33
+if not container.defined:
34
+ raise Exception("Container %s not defined!" % container_name)
35
+
36
+# Start the container
37
+if not container.start():
38
+ raise Exception("Failed to start the container")
39
+
40
+if not container.running or not container.state == "RUNNING":
41
+ raise Exception('Container %s is not running, configuration process aborted ' % container_name)
42
+
43
+# Wait for connectivity
44
+print ("Waiting for container connectivity to start configuration sequence")
45
+if not container.get_ips(timeout=30):
46
+ raise Exception("Timeout while waiting for container")
47
+
48
+# Run the required activities now
49
+# 1. Create the builder user
50
+print ("1. Adding user %s" % os.environ['BUILDER_NAME'])
51
+run_command(["useradd", "-m", os.environ['BUILDER_NAME']])
52
+
53
+# Fetch package dependencies for the build
54
+print ("2. Installing package dependencies within LXC container")
55
+run_command(["apt-get", "update", "-y"])
56
+run_command(["apt-get", "install", "-y", "sudo"])
57
+run_command(["apt-get", "install", "-y", "wget"])
58
+run_command(["apt-get", "install", "-y", "bash"])
59
+run_command(["wget", "-T", "15", "-O", "~/.install-required-packages.sh", "https://raw.githubusercontent.com/netdata/netdata-demo-site/master/install-required-packages.sh"])
60
+run_command(["bash", "~/.install-required-packages.sh", "netdata", "--dont-wait", "--non-interactive"])
61
+
62
+# Download the source
63
+dest_archive="/home/%s/netdata-%s.tar.gz" % (os.environ['BUILDER_NAME'],os.environ['BUILD_VERSION'])
64
+release_url="https://github.com/netdata/netdata/releases/download/%s/netdata-%s.tar.gz" % (os.environ['BUILD_VERSION'], os.environ['BUILD_VERSION'])
65
+print ("3. Fetch netdata source (%s -> %s)" % (release_url, dest_archive))
66
+run_command(["sudo", "-u", os.environ['BUILDER_NAME'], "wget", "-T", "15", "--output-document=" + dest_archive, release_url])
67
+
68
+print ("4. Extracting directory contents to /home " + os.environ['BUILDER_NAME'])
69
+run_command(["sudo", "-u", os.environ['BUILDER_NAME'], "tar", "xf", dest_archive, "-C", "/home/" + os.environ['BUILDER_NAME']])
70
+
71
+print ('Done!')
.travis/package_management/deb/trigger_lxc_build.py
new
+58
@@ -0,0 +1,58 @@
1
+#!/usr/bin/env python3
2
+#
3
+# This script is responsible for running the RPM build on the running container
4
+#
5
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
6
+#
7
+# Author : Pavlos Emm. Katsoulakis <paul@netdata.cloud>
8
+
9
+import os
10
+import sys
11
+import lxc
12
+
13
+def run_command(command):
14
+ print ("Running command: %s" % command)
15
+ command_result = container.attach_wait(lxc.attach_run_command, command)
16
+
17
+ if command_result != 0:
18
+ raise Exception("Command failed with exit code %d" % command_result)
19
+
20
+print (sys.argv)
21
+if len(sys.argv) != 2:
22
+ print ('You need to provide a container name to get things started')
23
+ sys.exit(1)
24
+container_name=sys.argv[1]
25
+
26
+# Load the container, break if its not there
27
+print ("Starting up container %s" % container_name)
28
+container = lxc.Container(container_name)
29
+if not container.defined:
30
+ raise Exception("Container %s does not exist!" % container_name)
31
+
32
+# Check if the container is running, attempt to start it up in case its not running
33
+if not container.running or not container.state == "RUNNING":
34
+ print ('Container %s is not running, attempt to start it up' % container_name)
35
+
36
+ # Start the container
37
+ if not container.start():
38
+ raise Exception("Failed to start the container")
39
+
40
+ if not container.running or not container.state == "RUNNING":
41
+ raise Exception('Container %s is not running, configuration process aborted ' % container_name)
42
+
43
+# Wait for connectivity
44
+if not container.get_ips(timeout=30):
45
+ raise Exception("Timeout while waiting for container")
46
+
47
+print ("Setting up EMAIL and DEBFULLNAME variables required by the build tools")
48
+os.environ["EMAIL"] = "bot@netdata.cloud"
49
+os.environ["DEBFULLNAME"] = "Netdata builder"
50
+
51
+# Run the build process on the container
52
+print ("Starting DEB build process, running dh-make")
53
+new_version = os.environ["BUILD_VERSION"].replace('v', '')
54
+
55
+print ("Building the package")
56
+run_command(["sudo", "-u", os.environ['BUILDER_NAME'], "dpkg-buildpackage", "--host-arch", "amd64", "--target-arch", "amd64", "--post-clean", "--pre-clean", "--build=binary", "--release-by=\"Netdata Builder\"", "--build-by=\"Netdata Builder\""])
57
+
58
+print ('Done!')
.travis/package_management/functions.sh
new
+33
@@ -0,0 +1,33 @@
1
+# no-shebang-needed-its-a-library
2
+#
3
+# Utility functions for packaging in travis CI
4
+#
5
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
6
+#
7
+# Author : Pavlos Emm. Katsoulakis (paul@netdata.cloud)
8
+#shellcheck disable=SC2148
9
+set -e
10
+
11
+function detect_arch_from_commit {
12
+ case "${TRAVIS_COMMIT_MESSAGE}" in
13
+ "[Package AMD64"*)
14
+ export BUILD_ARCH="amd64"
15
+ ;;
16
+ "[Package i386"*)
17
+ export BUILD_ARCH="i386"
18
+ ;;
19
+ "[Package ALL"*)
20
+ export BUILD_ARCH="all"
21
+ ;;
22
+ "[Package arm64"*)
23
+ export BUILD_ARCH="arm64"
24
+ ;;
25
+
26
+ *)
27
+ echo "Unknown build architecture '${BUILD_ARCH}' provided"
28
+ exit 1
29
+ ;;
30
+ esac
31
+
32
+ echo "Detected build architecture ${BUILD_ARCH}"
33
+}
.travis/package_management/package_cloud_wrapper.sh
new
+48
@@ -0,0 +1,48 @@
1
+#!/usr/bin/env bash
2
+#
3
+# This is a tool to help removal of packages from packagecloud.io
4
+# It utilizes the package_cloud utility provided from packagecloud.io
5
+#
6
+# Depends on:
7
+# 1) package cloud gem (detects absence and installs it)
8
+#
9
+# Requires:
10
+# 1) PKG_CLOUD_TOKEN variable exported
11
+# 2) To properly install package_cloud when not found, it requires: ruby gcc gcc-c++ ruby-devel
12
+#
13
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
14
+#
15
+# Author : Pavlos Emm. Katsoulakis (paul@netdata.cloud)
16
+#shellcheck disable=SC2068,SC2145
17
+set -e
18
+PKG_CLOUD_CONFIG="$HOME/.package_cloud_configuration.cfg"
19
+
20
+# If we are not in netdata git repo, at the top level directory, fail
21
+TOP_LEVEL=$(basename "$(git rev-parse --show-toplevel)")
22
+CWD=$(git rev-parse --show-cdup)
23
+if [ -n "$CWD" ] || [ ! "${TOP_LEVEL}" == "netdata" ]; then
24
+ echo "Run as .travis/package_management/$(basename "$0") from top level directory of netdata git repository"
25
+ echo "Docker build process aborted"
26
+ exit 1
27
+fi
28
+
29
+# Install dependency if not there
30
+if ! command -v package_cloud > /dev/null 2>&1; then
31
+ echo "No package cloud gem found, installing"
32
+ gem install -V package_cloud || (echo "Package cloud installation failed. you might want to check if required dependencies are there (ruby gcc gcc-c++ ruby-devel)" && exit 1)
33
+else
34
+ echo "Found package_cloud gem, continuing"
35
+fi
36
+
37
+# Check for required token and prepare config
38
+if [ -z "${PKG_CLOUD_TOKEN}" ]; then
39
+ echo "Please set PKG_CLOUD_TOKEN to be able to use ${0}"
40
+ exit 1
41
+fi
42
+echo "{\"url\":\"https://packagecloud.io\",\"token\":\"${PKG_CLOUD_TOKEN}\"}" > "${PKG_CLOUD_CONFIG}"
43
+
44
+echo "Executing package_cloud with config ${PKG_CLOUD_CONFIG} and parameters $@"
45
+package_cloud $@ --config="${PKG_CLOUD_CONFIG}"
46
+
47
+rm -rf "${PKG_CLOUD_CONFIG}"
48
+echo "Done!"
.travis/package_management/prepare_packages.sh
new
+56
@@ -0,0 +1,56 @@
1
+#!/usr/bin/env bash
2
+#
3
+# Utility that gathers generated packages,
4
+# puts them together in a local folder for deploy facility to pick up
5
+#
6
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
7
+#
8
+# Author : Pavlos Emm. Katsoulakis (paul@netdata.cloud)
9
+#shellcheck disable=SC2068
10
+set -e
11
+
12
+# If we are not in netdata git repo, at the top level directory, fail
13
+TOP_LEVEL=$(basename "$(git rev-parse --show-toplevel)")
14
+CWD=$(git rev-parse --show-cdup)
15
+if [ -n "$CWD" ] || [ ! "${TOP_LEVEL}" == "netdata" ]; then
16
+ echo "Run as .travis/package_management/$(basename "$0") from top level directory of netdata git repository"
17
+ echo "Package preparation aborted"
18
+ exit 1
19
+fi
20
+
21
+export LXC_ROOT="/var/lib/lxc"
22
+
23
+# Go through the containers created for packaging and pick up all generated packages
24
+CREATED_CONTAINERS=$(ls -A "${LXC_ROOT}")
25
+for d in ${CREATED_CONTAINERS[@]}; do
26
+ echo "Picking up packaging contents from ${d}"
27
+
28
+ # Pick up any RPMS from builder
29
+ RPM_BUILD_PATH="${LXC_ROOT}/${d}/rootfs/home/${BUILDER_NAME}/rpmbuild"
30
+ echo "Checking folder ${RPM_BUILD_PATH} for RPMS and SRPMS"
31
+
32
+ if [ -d "${RPM_BUILD_PATH}/RPMS" ]; then
33
+ echo "Copying any RPMS in '${RPM_BUILD_PATH}', copying over the following:"
34
+ ls -ltrR "${RPM_BUILD_PATH}/RPMS"
35
+ [[ -d "${RPM_BUILD_PATH}/RPMS/x86_64" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/x86_64/* "${PACKAGES_DIRECTORY}"
36
+ [[ -d "${RPM_BUILD_PATH}/RPMS/i386" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/i386/* "${PACKAGES_DIRECTORY}"
37
+ [[ -d "${RPM_BUILD_PATH}/RPMS/i686" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/i686/* "${PACKAGES_DIRECTORY}"
38
+ fi
39
+
40
+ if [ -d "${RPM_BUILD_PATH}/SRPMS" ]; then
41
+ echo "Copying any SRPMS in '${RPM_BUILD_PATH}', copying over the following:"
42
+ ls -ltrR "${RPM_BUILD_PATH}/SRPMS"
43
+ [[ -d "${RPM_BUILD_PATH}/SRPMS/x86_64" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/x86_64/* "${PACKAGES_DIRECTORY}"
44
+ [[ -d "${RPM_BUILD_PATH}/SRPMS/i386" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/i386/* "${PACKAGES_DIRECTORY}"
45
+ [[ -d "${RPM_BUILD_PATH}/SRPMS/i686" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/i686/* "${PACKAGES_DIRECTORY}"
46
+ fi
47
+
48
+ # Pick up any DEBs from builder
49
+ DEB_BUILD_PATH="${d}/home/${BUILDER_NAME}/build-area"
50
+ echo "Checking folder ${DEB_BUILD_PATH} for DEB packages"
51
+ #TODO: During debian clean up we 'll fill this up
52
+
53
+done
54
+
55
+chmod -R 777 "${PACKAGES_DIRECTORY}"
56
+echo "Packaging contents ready to ship!"
.travis/package_management/rpm/configure_lxc_environment.py
new
+89
@@ -0,0 +1,89 @@
1
+#!/usr/bin/env python3
2
+#
3
+# Prepare the build environment within the container
4
+# The script attaches to the running container and does the following:
5
+# 1) Create the container
6
+# 2) Start the container up
7
+# 3) Create the builder user
8
+# 4) Prepare the environment for RPM build
9
+#
10
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
11
+#
12
+# Author : Pavlos Emm. Katsoulakis <paul@netdata.cloud>
13
+
14
+import os
15
+import sys
16
+import lxc
17
+
18
+def run_command(command):
19
+ print ("Running command: %s" % command)
20
+ command_result = container.attach_wait(lxc.attach_run_command, command)
21
+
22
+ if command_result != 0:
23
+ raise Exception("Command failed with exit code %d" % command_result)
24
+
25
+if len(sys.argv) != 2:
26
+ print ('You need to provide a container name to get things started')
27
+ sys.exit(1)
28
+container_name=sys.argv[1]
29
+
30
+# Setup the container object
31
+print ("Defining container %s" % container_name)
32
+container = lxc.Container(container_name)
33
+if not container.defined:
34
+ raise Exception("Container %s not defined!" % container_name)
35
+
36
+# Start the container
37
+if not container.start():
38
+ raise Exception("Failed to start the container")
39
+
40
+if not container.running or not container.state == "RUNNING":
41
+ raise Exception('Container %s is not running, configuration process aborted ' % container_name)
42
+
43
+# Wait for connectivity
44
+print ("Waiting for container connectivity to start configuration sequence")
45
+if not container.get_ips(timeout=30):
46
+ raise Exception("Timeout while waiting for container")
47
+
48
+# Run the required activities now
49
+# Create the builder user
50
+print ("1. Adding user %s" % os.environ['BUILDER_NAME'])
51
+run_command(["useradd", "-m", os.environ['BUILDER_NAME']])
52
+
53
+# Fetch package dependencies for the build
54
+print ("2. Installing package dependencies within LXC container")
55
+if str(os.environ["REPO_TOOL"]).count("zypper") == 1:
56
+ run_command([os.environ["REPO_TOOL"], "clean", "-a"])
57
+ run_command([os.environ["REPO_TOOL"], "--no-gpg-checks", "update", "-y"])
58
+else:
59
+ run_command([os.environ["REPO_TOOL"], "update", "-y"])
60
+
61
+run_command([os.environ["REPO_TOOL"], "install", "-y", "sudo"])
62
+run_command([os.environ["REPO_TOOL"], "install", "-y", "wget"])
63
+run_command([os.environ["REPO_TOOL"], "install", "-y", "bash"])
64
+run_command(["wget", "-T", "15", "-O", "~/.install-required-packages.sh", "https://raw.githubusercontent.com/netdata/netdata-demo-site/master/install-required-packages.sh"])
65
+run_command(["bash", "~/.install-required-packages.sh", "netdata", "--dont-wait", "--non-interactive"])
66
+
67
+print ("3. Setting up macros")
68
+run_command(["sudo", "-u", os.environ['BUILDER_NAME'], "/bin/echo", "'%_topdir %(echo /home/" + os.environ['BUILDER_NAME'] + ")/rpmbuild' > /home/" + os.environ['BUILDER_NAME'] + "/.rpmmacros"])
69
+
70
+print ("4. Create rpmbuild directory")
71
+run_command(["sudo", "-u", os.environ['BUILDER_NAME'], "mkdir", "-p", "/home/" + os.environ['BUILDER_NAME'] + "/rpmbuild/BUILD"])
72
+run_command(["sudo", "-u", os.environ['BUILDER_NAME'], "mkdir", "-p", "/home/" + os.environ['BUILDER_NAME'] + "/rpmbuild/RPMS"])
73
+run_command(["sudo", "-u", os.environ['BUILDER_NAME'], "mkdir", "-p", "/home/" + os.environ['BUILDER_NAME'] + "/rpmbuild/SOURCES"])
74
+run_command(["sudo", "-u", os.environ['BUILDER_NAME'], "mkdir", "-p", "/home/" + os.environ['BUILDER_NAME'] + "/rpmbuild/SPECS"])
75
+run_command(["sudo", "-u", os.environ['BUILDER_NAME'], "mkdir", "-p", "/home/" + os.environ['BUILDER_NAME'] + "/rpmbuild/SRPMS"])
76
+run_command(["sudo", "-u", os.environ['BUILDER_NAME'], "ls", "-ltrR", "/home/" + os.environ['BUILDER_NAME'] + "/rpmbuild"])
77
+
78
+# Download the source
79
+dest_archive="/home/%s/rpmbuild/SOURCES/netdata-%s.tar.gz" % (os.environ['BUILDER_NAME'],os.environ['BUILD_VERSION'])
80
+release_url="https://github.com/netdata/netdata/releases/download/%s/netdata-%s.tar.gz" % (os.environ['BUILD_VERSION'], os.environ['BUILD_VERSION'])
81
+print ("5. Fetch netdata source into the repo structure(%s -> %s)" % (release_url, dest_archive))
82
+run_command(["sudo", "-u", os.environ['BUILDER_NAME'], "wget", "-T", "15", "--output-document=" + dest_archive, release_url])
83
+
84
+# Extract the spec file in place
85
+print ("6. Extract spec file from the source")
86
+spec_file="/home/%s/rpmbuild/SPECS/netdata.spec" % os.environ['BUILDER_NAME']
87
+run_command(["sudo", "-u", os.environ['BUILDER_NAME'], "tar", "--to-command=cat > %s" % spec_file, "-xvf", dest_archive, "netdata-%s/netdata.spec" % os.environ['BUILD_VERSION']])
88
+
89
+print ('Done!')
.travis/package_management/rpm/trigger_lxc_build.py
new
+62
@@ -0,0 +1,62 @@
1
+#!/usr/bin/env python3
2
+#
3
+# This script is responsible for running the RPM build on the running container
4
+#
5
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
6
+#
7
+# Author : Pavlos Emm. Katsoulakis <paul@netdata.cloud>
8
+
9
+import os
10
+import sys
11
+import lxc
12
+
13
+def run_command(command):
14
+ print ("Running command: %s" % command)
15
+ command_result = container.attach_wait(lxc.attach_run_command, command)
16
+
17
+ if command_result != 0:
18
+ raise Exception("Command failed with exit code %d" % command_result)
19
+
20
+print (sys.argv)
21
+if len(sys.argv) != 2:
22
+ print ('You need to provide a container name to get things started')
23
+ sys.exit(1)
24
+container_name=sys.argv[1]
25
+
26
+# Load the container, break if its not there
27
+print ("Starting up container %s" % container_name)
28
+container = lxc.Container(container_name)
29
+if not container.defined:
30
+ raise Exception("Container %s does not exist!" % container_name)
31
+
32
+# Check if the container is running, attempt to start it up in case its not running
33
+if not container.running or not container.state == "RUNNING":
34
+ print ('Container %s is not running, attempt to start it up' % container_name)
35
+
36
+ # Start the container
37
+ if not container.start():
38
+ raise Exception("Failed to start the container")
39
+
40
+ if not container.running or not container.state == "RUNNING":
41
+ raise Exception('Container %s is not running, configuration process aborted ' % container_name)
42
+
43
+# Wait for connectivity
44
+if not container.get_ips(timeout=30):
45
+ raise Exception("Timeout while waiting for container")
46
+
47
+print ("Adding builder specific dependencies to the LXC container")
48
+run_command([os.environ["REPO_TOOL"], "install", "-y", "rpm-build"])
49
+run_command([os.environ["REPO_TOOL"], "install", "-y", "rpm-devel"])
50
+run_command([os.environ["REPO_TOOL"], "install", "-y", "rpmlint"])
51
+run_command([os.environ["REPO_TOOL"], "install", "-y", "make"])
52
+run_command([os.environ["REPO_TOOL"], "install", "-y", "python"])
53
+run_command([os.environ["REPO_TOOL"], "install", "-y", "bash"])
54
+run_command([os.environ["REPO_TOOL"], "install", "-y", "diffutils"])
55
+run_command([os.environ["REPO_TOOL"], "install", "-y", "patch"])
56
+run_command([os.environ["REPO_TOOL"], "install", "-y", "rpmdevtools"])
57
+
58
+# Run the build process on the container
59
+print ("Starting RPM build process")
60
+run_command(["sudo", "-u", os.environ['BUILDER_NAME'], "rpmbuild", "-ba", "--rebuild", "/home/%s/rpmbuild/SPECS/netdata.spec" % os.environ['BUILDER_NAME']])
61
+
62
+print ('Done!')
.travis/package_management/yank_stale_rpm.sh
new
+35
@@ -0,0 +1,35 @@
1
+#!/usr/bin/env bash
2
+#
3
+# This script is responsible for the removal of stale RPM/DEB files.
4
+# It runs on the pre-deploy step and takes care of the removal of the files
5
+# prior to the upload of the freshly built ones
6
+#
7
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
8
+#
9
+# Author : Pavlos Emm. Katsoulakis (paul@netdata.cloud)
10
+#shellcheck disable=SC2010,SC2068
11
+set -e
12
+
13
+# If we are not in netdata git repo, at the top level directory, fail
14
+TOP_LEVEL=$(basename "$(git rev-parse --show-toplevel)")
15
+CWD=$(git rev-parse --show-cdup)
16
+if [ -n "$CWD" ] || [ ! "${TOP_LEVEL}" == "netdata" ]; then
17
+ echo "Run as .travis/package_management/$(basename "$0") from top level directory of netdata git repository"
18
+ echo "Package yanking cancelled"
19
+ exit 1
20
+fi
21
+
22
+PACKAGES_DIR="$1"
23
+DISTRO="$2"
24
+PACKAGES_LIST="$(ls -AR "${PACKAGES_DIR}" | grep '\.rpm')"
25
+
26
+if [ ! -d "${PACKAGES_DIR}" ] || [ -z "${PACKAGES_LIST}" ]; then
27
+ echo "Folder ${PACKAGES_DIR} does not seem to be a valid directory or is empty. No packages to check for yanking"
28
+ exit 1
29
+fi
30
+
31
+for pkg in ${PACKAGES_LIST[@]}; do
32
+ echo "Attempting yank on ${pkg}.."
33
+ .travis/package_management/package_cloud_wrapper.sh yank "${TRAVIS_REPO_SLUG}/${DISTRO}" "${pkg}" || echo "Nothing to yank or error on ${pkg}"
34
+done
35
+
Makefile.am
+1
-4
@@ -20,13 +20,9 @@ CLEANFILES= \
20
21
EXTRA_DIST = \
22
.gitignore \
23
- .codacy.yml \
24
- .codeclimate.yml \
23
.csslintrc \
24
.eslintignore \
25
.eslintrc \
28
- .lgtm.yml \
29
- .travis \
26
.github/CODEOWNERS \
27
build/m4/jemalloc.m4 \
28
build/m4/ax_c___atomic.m4 \
@@ -55,6 +51,7 @@ SUBDIRS = \
51
$(NULL)
52
53
dist_noinst_DATA= \
54
+ CHANGELOG.md \
55
cppcheck.sh \
56
configs.signatures \
57
contrib \