@cryptotaxi247 / netdata-1 / commits / 079d71d6f

netdata/packaging: Binary distributions - clean up .DEB package generation process (#6465)

* netdata/packaging: [ci skip] reinstate dev branch to help testing * netdata/packaging: [ci skip] Yes, make the right package choice on json-c for debian based distros.. * netdata/packaging: [ci skip] two more packages that differentiate on debian/ubuntu............... * netdata/packaging: [ci skip] Add more package dependencies around building * netdata/packaging: [ci skip] Introduce changelog generation technique for .DEB 1) Remove contrib/debian/changelog from gitignore, we want it in 2) Add variables to monitor latest release version and latest release date from git 3) Add a templated contrib/debian/changelog, that will be the starting point for our changelog generation. Either on stable release or nightly releases, we will be generating the logs since the previous release, to manage the content. Note: We might eventually take the effort to produce one large changelog and append each time the extra stuff 4) Add the steps to prepare the source and the changelog for building and execute the build At this stage things should still be broken, but we should have come closer to a ready-to-build-the-package environment per distro * netdata/packaging: [ci skip] Extract to specified directory, otherwise we dump it on a privileged folder using an unprivileged user (builder) Also, a fix on sed command, missed the s keyword on the string replacement * netdata/packaging: [ci skip] Should be in-place replacement, use -i instead of -e * netdata/packaging: [ci skip] wrong parameter usage * netdata/packaging: [ci skip] factor out the two build commands to run as a single bundle. refactor trigger deb build script respectively * netdata/packaging: [ci skip] Be sure to copy over the build script to the build path * netdata/packaging: [ci skip] OCD - unnecessary s added * netdata/packaging: [ci skip] Changelog generation should take place on the repo side, then copy it over to the destination (If that doesnt work well, will just provide a static link for changelog * netdata/packaging: [ci skip] missed the ignore branch option Also, a nit on the gbp command * netdata/packaging: [ci skip] link to the original path, then create symlink on build folder Also, fix --since option * netdata/packaging: [ci skip] Run replacement of changelog fields on the host, not on the container * netdata/packaging: [ci skip] fix contrib path, you are no longer reading from container path * netdata/packaging: [ci skip] fix version * netdata/packaging: [ci skip] add dep * netdata/packaging: [ci skip] fix changelog template * netdata/packaging: [ci skip] Fix debian package build, we were looping through forever. Exclude contrib all together * netdata/packaging: [ci skip] Add logic to gather all package related content from the container folder and push it for publishing * netdata/packaging: [ci skip] fixes and nits 1) use the branch for the beta deployment 2) change the cp pattern * netdata/packaging:[ci skip] add more debug info * netdata/packaging: [ci skip] Adjust LXC directory permissions, so that the rest of the workflow can pull through * netdata/packaging: [ci skip] Using expressions within double quotes doesnt work, use quotes only on the variable to avoid syntax warnings from shellchecking * netdata/packaging: [ci skip] Update build deps * netdata/packaging: Add runtime deps (hopefully) * netdata/packaging: [ci skip] remove old distro * netdata/packaging: [ci skip] dont let that space there, breaks the syntax * netdata/packaging: [ci skip] missed one * netdata/packaging: [ci skip] yank yank yank * netdata/packaging: [ci skip] You missed www over there * netdata/packaging: [ci skip] Remove broken distros (no container images). We will revisit the way to support multiple distros in a different way, lxc failed us * netdata/packaging: Pull in go.d plugin, also fix chmod command to just pass the folder since with the -R doesnt require anything else * netdata/packaging: [ci skip] Get cups, freeipmi and nfacct on * netdata/packaging: [ci skip] pull different package in * netdata/packaging: [ci skip] Install dependencies on the build machine * netdata/packaging: [ci skip] dont do post-clean, not on all versions * netdata/packaging: [ci skip] move go install seperately * netdata/packaging: [ci skip] Towards a more appropriate go.d plugin install * netdata/packaging: [ci skip] fix permissions setting * netdata/packaging: [ci skip] fix freeipmi package install * netdata/packaging: [ci skip] Fix build arch setting on the build command, also remove pre-clean, it breaks on debian/jessie * netdata/packaging: [ci skip] fix postinst actions, we use libexec * netdata/packaging: adjust www and go.d permissions * netdata/packaging: [ci skip] Handle the different build tool version cases, also take care some dependency issues on older version jessie * netdata/packaging: [ci skip] adjust the control file based on the distro * netdata/packaging: [ci skip] Move control file preparation prior to packaging the source * netdata/packaging: [ci skip] Adjust versioning scheme for the nightlies * netdata/packaging: [ci skip] Let RPM be built too for testing * netdata/packaging: [ci skip] Factor out version calculation * netdata/packaging: [ci skip] Attempt to modify version properly * netdata/packaging: [ci skip] Update documentationg regarding .DEB files * netdata/packaging: [ci skip] autogen not available everywhere, also fix json-c deps * netdata/packaging: Remove development settings, also remove beta deployment as its not needed

Paul Emm. Katsoulakis committed Jul 15, 2019 at 22:00 UTC 079d71d6fbc81fdab17dbdf5c07669fc8ad6cc4d

16 files changed +351 -117

.gitignore

-2

@@ -139,8 +139,6 @@ cmake_install.cmake

139 .DS_Store

140 webcopylocal*

141

142 -contrib/debian/changelog

143 -

142 # converted diagrams

143 diagrams/*.png

144 diagrams/*.svg

.travis.yml

+5 -54

@@ -24,6 +24,8 @@ install:

24 - export NOTIF_CHANNEL="automation-beta"

25 - if [ "${TRAVIS_REPO_SLUG}" = "netdata/netdata" ]; then export NOTIF_CHANNEL="automation"; fi;

26 - export BUILD_VERSION="$(cat packaging/version | cut -d'-' -f1)"

27 +- export LATEST_RELEASE_VERSION="$(cat packaging/version | cut -d'-' -f1)"

28 +- export LATEST_RELEASE_DATE="$(git log -1 --format=%aD "${LATEST_RELEASE_VERSION}" | cat)"

29 - if [[ "${TRAVIS_COMMIT_MESSAGE}" = *"[Build latest]"* ]]; then export BUILD_VERSION="$(cat packaging/version | cut -d'-' -f1,2 | sed -e 's/-/./g').latest"; fi;

30 - export DEPLOY_REPO="netdata" # Default production packaging repository

31 - if [[ "${TRAVIS_COMMIT_MESSAGE}" = *"[Build latest]"* ]]; then export DEPLOY_REPO="netdata-edge"; fi;

@@ -69,8 +71,6 @@ stages:

72 - name: "Package ubuntu/bionic"

72 -- name: "Package ubuntu/artful"

75 # Debian

76 - name: "Package debian/buster"

@@ -79,8 +79,6 @@ stages:

80 - name: "Package debian/jessie"

82 -- name: "Package debian/wheezy"

83 # Build RPM packages under special conditions

84 # Enterprise linux (Covers CentOS, Redhat, Amazon linux)

@@ -129,7 +127,6 @@ stages:

127 before_deploy:

128 - .travis/package_management/yank_stale_rpm.sh "${PACKAGES_DIRECTORY}" "${BUILD_STRING}" || echo "No stale RPM found"

129 deploy:

132 - # Beta packages deployment

130 - provider: packagecloud

131 repository: "${DEPLOY_REPO}"

132 username: "${PACKAGING_USER}"

@@ -142,19 +139,6 @@ stages:

139 repo: ${TRAVIS_REPO_SLUG}

140 branch: "master"

141 condition: -d "${PACKAGES_DIRECTORY}"

145 - # Production release packages deployment

146 - - provider: packagecloud

147 - repository: "netdata"

148 - username: "netdata"

149 - token: "${PKG_CLOUD_TOKEN}"

150 - dist: "${BUILD_STRING}"

151 - local_dir: "${PACKAGES_DIRECTORY}"

152 - skip_cleanup: true

153 - on:

154 - # Only deploy on ${USER}/netdata, master branch, when packages directory is created

155 - repo: "netdata/netdata"

156 - branch: "master"

157 - condition: -d "${PACKAGES_DIRECTORY}"

142 after_deploy:

143 - if [ -n "${BUILDER_NAME}" ]; then rm -rf /home/${BUILDER_NAME}/* && echo "Cleared /home/${BUILDER_NAME} directory" || echo "Failed to clean /home/${BUILDER_NAME} directory"; fi;

144 - if [ -d "${PACKAGES_DIRECTORY}" ]; then rm -rf "${PACKAGES_DIRECTORY}"; fi;

@@ -165,7 +149,7 @@ stages:

149 - stage: &_DEB_TEMPLATE

150 name: "Build & Publish DEB package"

151 before_install:

168 - - sudo apt-get install -y wget lxc lxc-templates

152 + - sudo apt-get install -y wget lxc lxc-templates dh-make git-buildpackage build-essential libdistro-info-perl

153 - source tests/installer/slack.sh

154 before_script:

155 - post_message "TRAVIS_MESSAGE" "Starting package preparation and publishing for ${BUILD_STRING}.${BUILD_ARCH}" "${NOTIF_CHANNEL}"

@@ -177,7 +161,8 @@ stages:

161 - echo "packaging/version:" && cat packaging/version

162 - echo "Creating LXC environment for the build" && sudo -E .travis/package_management/create_lxc_for_build.sh

163 - echo "Building package in container" && sudo -E .travis/package_management/build_package_in_container.sh

180 - - sudo chmod -R 755 "/var/lib/lxc"

164 + - sudo chown -R root:travis "/var/lib/lxc"

165 + - sudo chmod -R 750 "/var/lib/lxc"

166 - echo "Preparing DEB packaging contents for upload" && sudo -E .travis/package_management/prepare_packages.sh

167 git:

168 depth: false

@@ -185,7 +170,6 @@ stages:

170 before_deploy:

171 - .travis/package_management/yank_stale_rpm.sh "${PACKAGES_DIRECTORY}" "${BUILD_STRING}" || echo "No stale DEB found"

172 deploy:

188 - # Beta packages deployment

173 - provider: packagecloud

174 repository: "${DEPLOY_REPO}"

175 username: "${PACKAGING_USER}"

@@ -198,19 +182,6 @@ stages:

182 repo: ${TRAVIS_REPO_SLUG}

183 branch: "master"

184 condition: -d "${PACKAGES_DIRECTORY}"

201 - # Production release packages deployment

202 - - provider: packagecloud

203 - repository: "netdata"

204 - username: "netdata"

205 - token: "${PKG_CLOUD_TOKEN}"

206 - dist: "${BUILD_STRING}"

207 - local_dir: "${PACKAGES_DIRECTORY}"

208 - skip_cleanup: true

209 - on:

210 - # Only deploy on ${USER}/netdata, master branch, when build-area directory is created

211 - repo: "netdata/netdata"

212 - branch: "master"

213 - condition: -d "${PACKAGES_DIRECTORY}"

185 after_deploy:

186 - if [ -n "${BUILDER_NAME}" ]; then rm -rf /home/${BUILDER_NAME}/* && echo "Cleared /home/${BUILDER_NAME} directory" || echo "Failed to clean /home/${BUILDER_NAME} directory"; fi;

187 - if [ -d "${PACKAGES_DIRECTORY}" ]; then rm -rf "${PACKAGES_DIRECTORY}"; fi;

@@ -422,16 +393,6 @@ jobs:

393

394

395

425 - - stage:

426 - <<: *_DEB_TEMPLATE

427 - stage: "Package ubuntu/artful"

428 - env:

429 - - BUILDER_NAME="builder" BUILD_DISTRO="ubuntu" BUILD_RELEASE="artful" BUILD_STRING="ubuntu/artful"

430 - - PACKAGE_TYPE="deb" REPO_TOOL="apt-get"

431 - - ALLOW_SOFT_FAILURE_HERE=true

432 -

433 -

434 -

396 # Debian distros build

397 - stage:

398 <<: *_DEB_TEMPLATE

@@ -463,16 +424,6 @@ jobs:

424

425

426

466 - - stage:

467 - <<: *_DEB_TEMPLATE

468 - stage: "Package debian/wheezy"

469 - env:

470 - - BUILDER_NAME="builder" BUILD_DISTRO="debian" BUILD_RELEASE="wheezy" BUILD_STRING="debian/wheezy"

471 - - PACKAGE_TYPE="deb" REPO_TOOL="apt-get"

472 - - ALLOW_SOFT_FAILURE_HERE=true

473 -

474 -

475 -

427 # Enterprise linux builds (Centos, Redhat, Amazon linux (el/6))

428 #

429 - stage:

.travis/package_management/build.sh new

+31

@@ -0,0 +1,31 @@

1 +#!/usr/bin/env bash

2 +

3 +UNPACKAGED_NETDATA_PATH="$1"

4 +LATEST_RELEASE_VERSION="$2"

5 +

6 +if [ -z "${LATEST_RELEASE_VERSION}" ]; then

7 + echo "Parameter 'LATEST_RELEASE_VERSION' not defined"

8 + exit 1

9 +fi

10 +

11 +if [ -z "${UNPACKAGED_NETDATA_PATH}" ]; then

12 + echo "Parameter 'UNPACKAGED_NETDATA_PATH' not defined"

13 + exit 1

14 +fi

15 +

16 +echo "Running changelog generation mechanism since ${LATEST_RELEASE_VERSION}"

17 +

18 +echo "Entering ${UNPACKAGED_NETDATA_PATH}"

19 +cd "${UNPACKAGED_NETDATA_PATH}"

20 +

21 +echo "Linking debian -> contrib/debian"

22 +ln -sf contrib/debian debian

23 +

24 +echo "Executing dpkg-buildpackage"

25 +if dpkg-buildpackage --version 2> /dev/null | grep -q "1.18"; then

26 + dpkg-buildpackage --post-clean --pre-clean --build=binary

27 +else

28 + dpkg-buildpackage -b

29 +fi

30 +

31 +echo "DEB build script completed!"

.travis/package_management/common.py

+30 -3

@@ -8,6 +8,21 @@ import lxc

8 import subprocess

9 import os

11 +def fetch_version(orig_build_version):

12 + tag = None

13 + friendly_version = ""

14 +

15 + # TODO: Checksum validations

16 + if str(orig_build_version).count(".latest") == 1:

17 + version_list=str(orig_build_version).replace('v', '').split('.')

18 + friendly_version='.'.join(version_list[0:2]) + "." + version_list[3]

19 + else:

20 + friendly_version = orig_build_version.replace('v', '')

21 + tag = friendly_version # Go to stable tag

22 + print("Version set to %s from %s" % (friendly_version, orig_build_version))

23 +

24 + return friendly_version, tag

25 +

26 def replace_tag(tag_name, spec, new_tag_content):

27 print("Fixing tag %s in %s" % (tag_name, spec))

@@ -51,24 +66,36 @@ def install_common_dependendencies(container):

66 run_command(container, [os.environ["REPO_TOOL"], "clean", "-a"])

67 run_command(container, [os.environ["REPO_TOOL"], "--no-gpg-checks", "update", "-y"])

68 run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "json-glib-devel"])

69 + run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "freeipmi-devel"])

70 + run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "cups-devel"])

72 elif str(os.environ["REPO_TOOL"]).count("yum") == 1:

56 - run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "json-c-devel"])

73 run_command(container, [os.environ["REPO_TOOL"], "clean", "all"])

74 run_command(container, [os.environ["REPO_TOOL"], "update", "-y"])

75 + run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "json-c-devel"])

76 + run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "freeipmi-devel"])

77 + run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "cups-devel"])

78 +

79 if os.environ["BUILD_STRING"].count("el/7") == 1 and os.environ["BUILD_ARCH"].count("i386") == 1:

80 print ("Skipping epel-release install for %s-%s" % (os.environ["BUILD_STRING"], os.environ["BUILD_ARCH"]))

81 else:

82 run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "epel-release"])

83 + elif str(os.environ["REPO_TOOL"]).count("apt-get") == 1:

84 + run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libipmimonitoring-dev"])

85 + run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libjson-c-dev"])

86 + run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libcups2-dev"])

87 else:

88 + run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "cups-devel"])

89 + run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "freeipmi-devel"])

90 run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "json-c-devel"])

91 run_command(container, [os.environ["REPO_TOOL"], "update", "-y"])

93 + if os.environ["BUILD_STRING"].count("el/6") < 0:

94 + run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "autogen"])

95 +

96 run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "sudo"])

97 run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "wget"])

98 run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "bash"])

70 - run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "freeipmi-devel"])

71 - run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "cups-devel"])

100 def prepare_version_source(dest_archive, pkg_friendly_version, tag=None):

101 print(".0 Preparing local implementation tarball for version %s" % pkg_friendly_version)

.travis/package_management/configure_deb_lxc_environment.py

+28 -11

@@ -39,6 +39,8 @@ print("Waiting for container connectivity to start configuration sequence")

39 if not container.get_ips(timeout=30):

40 raise Exception("Timeout while waiting for container")

42 +build_path = "/home/%s" % os.environ['BUILDER_NAME']

43 +

44 # Run the required activities now

45 # 1. Create the builder user

46 print("1. Adding user %s" % os.environ['BUILDER_NAME'])

@@ -48,28 +50,43 @@ common.run_command(container, ["useradd", "-m", os.environ['BUILDER_NAME']])

50 print("2. Installing package dependencies within LXC container")

51 common.install_common_dependendencies(container)

53 +print("2.1 Install .DEB build support packages")

54 +common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "dpkg-dev"])

55 +common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libdistro-info-perl"])

56 +common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "dh-make"])

57 +common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "dh-systemd"])

58 +common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "dh-autoreconf"])

59 +common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "git-buildpackage"])

60 +

61 +print("2.2 Add more dependencies")

62 +common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libnetfilter-acct-dev"])

63 +common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libcups2-dev"])

64 +

65 print ("3. Run install-required-packages scriptlet")

52 -common.run_command(container, ["wget", "-T", "15", "-O", "/home/%s/.install-required-packages.sh" % (os.environ['BUILDER_NAME']), "https://raw.githubusercontent.com/netdata/netdata-demo-site/master/install-required-packages.sh"])

53 -common.run_command(container, ["bash", "/home/%s/.install-required-packages.sh" % (os.environ['BUILDER_NAME']), "netdata", "--dont-wait", "--non-interactive"])

66 +common.run_command(container, ["wget", "-T", "15", "-O", "%s/.install-required-packages.sh" % build_path, "https://raw.githubusercontent.com/netdata/netdata-demo-site/master/install-required-packages.sh"])

67 +common.run_command(container, ["bash", "%s/.install-required-packages.sh" % build_path, "netdata", "--dont-wait", "--non-interactive"])

69 friendly_version=""

70 dest_archive=""

71 download_url=""

72 tag = None

59 -

60 -# TODO: Checksum validations

61 -if str(os.environ['BUILD_VERSION']).count(".latest") == 1:

62 - version_list=str(os.environ['BUILD_VERSION']).replace('v', '').split('.')

63 - friendly_version='.'.join(version_list[0:3]) + "." + version_list[3]

64 -else:

65 - friendly_version = os.environ['BUILD_VERSION'].replace('v', '')

66 - tag = friendly_version # Go to stable tag

73 +friendly_version, tag = common.fetch_version(os.environ['BUILD_VERSION'])

75 tar_file="%s/netdata-%s.tar.gz" % (os.path.dirname(dest_archive), friendly_version)

77 print("5. I will be building version '%s' of netdata." % os.environ['BUILD_VERSION'])

71 -dest_archive="/home/%s/netdata-%s.tar.gz" % (os.environ['BUILDER_NAME'], friendly_version)

78 +dest_archive="%s/netdata-%s.tar.gz" % (build_path, friendly_version)

79 +

80 +if str(os.environ["BUILD_STRING"]).count("debian/jessie") == 1:

81 + print("5.1 We are building for Jessie, adjusting control file")

82 + common.run_command_in_host(['sudo', 'rm', 'contrib/debian/control'])

83 + common.run_command_in_host(['sudo', 'cp', 'contrib/debian/control.jessie', 'contrib/debian/control'])

85 common.prepare_version_source(dest_archive, friendly_version, tag=tag)

87 +print("6. Installing build.sh script to build path")

88 +common.run_command_in_host(['sudo', 'cp', '.travis/package_management/build.sh', "%s/%s/build.sh" % (os.environ['LXC_CONTAINER_ROOT'], build_path)])

89 +common.run_command_in_host(['sudo', 'chmod', '777', "%s/%s/build.sh" % (os.environ['LXC_CONTAINER_ROOT'], build_path)])

90 +common.run_command_in_host(['sudo', 'ln', '-sf', 'contrib/debian', 'debian'])

91 +

92 print("Done!")

.travis/package_management/configure_rpm_lxc_environment.py

+1 -9

@@ -79,15 +79,7 @@ dest_archive=""

79 download_url=""

80 spec_file="/home/%s/rpmbuild/SPECS/netdata.spec" % os.environ['BUILDER_NAME']

81 tag = None

82 -

83 -# TODO: Checksum validations

84 -if str(os.environ['BUILD_VERSION']).count(".latest") == 1:

85 - version_list=str(os.environ['BUILD_VERSION']).replace('v', '').split('.')

86 - rpm_friendly_version='.'.join(version_list[0:3]) + "." + version_list[3]

87 -else:

88 - rpm_friendly_version = os.environ['BUILD_VERSION'].replace('v', '')

89 - tag = rpm_friendly_version # Go to stable tag

90 -

82 +rpm_friendly_version, tag = common.fetch_version(os.environ['BUILD_VERSION'])

83 tar_file="%s/netdata-%s.tar.gz" % (os.path.dirname(dest_archive), rpm_friendly_version)

85 print("5. I will be building version '%s' of netdata." % os.environ['BUILD_VERSION'])

.travis/package_management/prepare_packages.sh

+29 -22

@@ -27,29 +27,36 @@ for d in ${CREATED_CONTAINERS[@]}; do

28 # Pick up any RPMS from builder

29 RPM_BUILD_PATH="${LXC_ROOT}/${d}/rootfs/home/${BUILDER_NAME}/rpmbuild"

30 - echo "Checking folder ${RPM_BUILD_PATH} for RPMS and SRPMS"

31 -

32 - if [ -d "${RPM_BUILD_PATH}/RPMS" ]; then

33 - echo "Copying any RPMS in '${RPM_BUILD_PATH}', copying over the following:"

34 - ls -ltrR "${RPM_BUILD_PATH}/RPMS"

35 - [[ -d "${RPM_BUILD_PATH}/RPMS/x86_64" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/x86_64/* "${PACKAGES_DIRECTORY}"

36 - [[ -d "${RPM_BUILD_PATH}/RPMS/i386" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/i386/* "${PACKAGES_DIRECTORY}"

37 - [[ -d "${RPM_BUILD_PATH}/RPMS/i686" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/i686/* "${PACKAGES_DIRECTORY}"

38 - fi

39 -

40 - if [ -d "${RPM_BUILD_PATH}/SRPMS" ]; then

41 - echo "Copying any SRPMS in '${RPM_BUILD_PATH}', copying over the following:"

42 - ls -ltrR "${RPM_BUILD_PATH}/SRPMS"

43 - [[ -d "${RPM_BUILD_PATH}/SRPMS/x86_64" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/x86_64/* "${PACKAGES_DIRECTORY}"

44 - [[ -d "${RPM_BUILD_PATH}/SRPMS/i386" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/i386/* "${PACKAGES_DIRECTORY}"

45 - [[ -d "${RPM_BUILD_PATH}/SRPMS/i686" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/i686/* "${PACKAGES_DIRECTORY}"

30 + if [ -d "${RPM_BUILD_PATH}" ]; then

31 + echo "Checking folder ${RPM_BUILD_PATH} for RPMS and SRPMS"

32 +

33 + if [ -d "${RPM_BUILD_PATH}/RPMS" ]; then

34 + echo "Copying any RPMS in '${RPM_BUILD_PATH}', copying over the following:"

35 + ls -ltrR "${RPM_BUILD_PATH}/RPMS"

36 + [[ -d "${RPM_BUILD_PATH}/RPMS/x86_64" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/x86_64/* "${PACKAGES_DIRECTORY}"

37 + [[ -d "${RPM_BUILD_PATH}/RPMS/i386" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/i386/* "${PACKAGES_DIRECTORY}"

38 + [[ -d "${RPM_BUILD_PATH}/RPMS/i686" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/i686/* "${PACKAGES_DIRECTORY}"

39 + fi

40 +

41 + if [ -d "${RPM_BUILD_PATH}/SRPMS" ]; then

42 + echo "Copying any SRPMS in '${RPM_BUILD_PATH}', copying over the following:"

43 + ls -ltrR "${RPM_BUILD_PATH}/SRPMS"

44 + [[ -d "${RPM_BUILD_PATH}/SRPMS/x86_64" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/x86_64/* "${PACKAGES_DIRECTORY}"

45 + [[ -d "${RPM_BUILD_PATH}/SRPMS/i386" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/i386/* "${PACKAGES_DIRECTORY}"

46 + [[ -d "${RPM_BUILD_PATH}/SRPMS/i686" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/i686/* "${PACKAGES_DIRECTORY}"

47 + fi

48 + else

49 + DEB_BUILD_PATH="${LXC_ROOT}/${d}/rootfs/home/${BUILDER_NAME}"

50 + echo "Checking folder ${DEB_BUILD_PATH} for DEB packages"

51 + if [ -d "${DEB_BUILD_PATH}" ]; then

52 + cp "${DEB_BUILD_PATH}"/netdata*.ddeb "${PACKAGES_DIRECTORY}" || echo "Could not copy any .ddeb files"

53 + cp "${DEB_BUILD_PATH}"/netdata*.deb "${PACKAGES_DIRECTORY}" || echo "Could not copy any .deb files"

54 + cp "${DEB_BUILD_PATH}"/netdata*.buildinfo "${PACKAGES_DIRECTORY}" || echo "Could not copy any .buildinfo files"

55 + cp "${DEB_BUILD_PATH}"/netdata*.changes "${PACKAGES_DIRECTORY}" || echo "Could not copy any .changes files"

56 + else

57 + echo "Folder ${DEB_BUILD_PATH} does not exist or not a directory, nothing to do for package preparation"

58 + fi

59 fi

47 -

48 - # Pick up any DEBs from builder

49 - DEB_BUILD_PATH="${d}/home/${BUILDER_NAME}/build-area"

50 - echo "Checking folder ${DEB_BUILD_PATH} for DEB packages"

51 - #TODO: During debian clean up we 'll fill this up

52 -

60 done

62 chmod -R 777 "${PACKAGES_DIRECTORY}"

.travis/package_management/trigger_deb_lxc_build.py

+26 -4

@@ -37,15 +37,37 @@ if not container.running or not container.state == "RUNNING":

37 if not container.get_ips(timeout=30):

38 raise Exception("Timeout while waiting for container")

40 +build_path = "/home/%s" % os.environ['BUILDER_NAME']

41 +

42 print("Setting up EMAIL and DEBFULLNAME variables required by the build tools")

43 os.environ["EMAIL"] = "bot@netdata.cloud"

44 os.environ["DEBFULLNAME"] = "Netdata builder"

46 # Run the build process on the container

45 -print("Starting DEB build process, running dh-make")

46 -new_version = os.environ["BUILD_VERSION"].replace('v', '')

47 +new_version, tag = common.fetch_version(os.environ['BUILD_VERSION'])

48 +print("Starting DEB build process for version %s" % new_version)

49 +

50 +netdata_tarball = "%s/netdata-%s.tar.gz" % (build_path, new_version)

51 +unpacked_netdata = netdata_tarball.replace(".tar.gz", "")

52 +

53 +print("Extracting tarball %s" % netdata_tarball)

54 +common.run_command(container, ["sudo", "-u", os.environ['BUILDER_NAME'], "tar", "xf", netdata_tarball, "-C", build_path])

55 +

56 +print("Fixing changelog tags")

57 +changelog_in_host = "contrib/debian/changelog"

58 +common.run_command_in_host(['sed', '-i', 's/PREVIOUS_PACKAGE_VERSION/%s-1/g' % os.environ["LATEST_RELEASE_VERSION"].replace("v", ""), changelog_in_host])

59 +common.run_command_in_host(['sed', '-i', 's/PREVIOUS_PACKAGE_DATE/%s/g' % os.environ["LATEST_RELEASE_DATE"], changelog_in_host])

60 +

61 +print("Executing gbp dch command..")

62 +common.run_command_in_host(['gbp', 'dch', '--release', '--ignore-branch', '--spawn-editor=snapshot', '--since=%s' % os.environ["LATEST_RELEASE_VERSION"], '--new-version=%s' % new_version])

63 +

64 +print("Copying over changelog to the destination machine")

65 +common.run_command_in_host(['sudo', 'cp', 'debian/changelog', "%s/%s/netdata-%s/contrib/debian/" % (os.environ['LXC_CONTAINER_ROOT'], build_path, new_version)])

66 +

67 +print("Running debian build script since %s" % os.environ["LATEST_RELEASE_VERSION"])

68 +common.run_command(container, ["sudo", "-u", os.environ['BUILDER_NAME'], "%s/build.sh" % build_path, unpacked_netdata, new_version])

48 -print("Building the package")

49 -common.run_command(container, ["sudo", "-u", os.environ['BUILDER_NAME'], "dpkg-buildpackage", "--host-arch", "amd64", "--target-arch", "amd64", "--post-clean", "--pre-clean", "--build=binary", "--release-by=\"Netdata Builder\"", "--build-by=\"Netdata Builder\""])

70 +print("Listing contents on build path")

71 +common.run_command(container, ["sudo", "-u", os.environ['BUILDER_NAME'], "ls", "-ltr", build_path])

73 print('Done!')

.travis/package_management/yank_stale_rpm.sh

+1 -1

@@ -21,7 +21,7 @@ fi

22 PACKAGES_DIR="$1"

23 DISTRO="$2"

24 -PACKAGES_LIST="$(ls -AR "${PACKAGES_DIR}" | grep '\.rpm')"

24 +PACKAGES_LIST="$(ls -AR "${PACKAGES_DIR}" | grep -e '\.rpm' -e '\.deb' -e '\.ddeb' )"

26 if [ ! -d "${PACKAGES_DIR}" ] || [ -z "${PACKAGES_LIST}" ]; then

27 echo "Folder ${PACKAGES_DIR} does not seem to be a valid directory or is empty. No packages to check for yanking"

contrib/debian/changelog new

@@ -0,0 +1,5 @@

1 +netdata (PREVIOUS_PACKAGE_VERSION) unstable; urgency=medium

2 +

3 + * Initial Release

4 +

5 +-- Netdata Builder <bot@netdata.cloud> PREVIOUS_PACKAGE_DATE

contrib/debian/control

+27 -1

@@ -4,7 +4,22 @@ Build-Depends: debhelper (>= 9),

4 dh-systemd (>= 1.5),

5 dpkg-dev (>= 1.13.19),

6 zlib1g-dev,

7 - uuid-dev

7 + uuid-dev,

8 + libuv1-dev,

9 + liblz4-dev,

10 + libjudy-dev,

11 + libssl-dev,

12 + libmnl-dev,

13 + libjson-c-dev,

14 + libcups2-dev,

15 + libipmimonitoring-dev,

16 + libnetfilter-acct-dev,

17 + autogen,

18 + autoconf,

19 + automake,

20 + pkg-config,

21 + curl,

22 + gcc

23 Section: net

24 Priority: optional

25 Maintainer: Costa Tsaousis <costa@tsaousis.gr>

@@ -16,6 +31,17 @@ Architecture: any

31 Depends: adduser,

32 libcap2-bin (>= 1:2.0),

33 lsb-base (>= 3.1-23.2),

34 + zlib1g,

35 + libuuid1,

36 + libuv1,

37 + liblz4-1,

38 + libjudydebian1,

39 + openssl,

40 + libmnl0,

41 + libjson-c3,

42 + cups,

43 + freeipmi,

44 + libnetfilter-acct1,

45 ${misc:Depends},

46 ${shlibs:Depends}

47 Description: real-time charts for system monitoring

contrib/debian/control.jessie new

+49

@@ -0,0 +1,49 @@

1 +Source: netdata

2 +Build-Depends: debhelper (>= 9),

3 + dh-autoreconf,

4 + dh-systemd (>= 1.5),

5 + dpkg-dev (>= 1.13.19),

6 + zlib1g-dev,

7 + uuid-dev,

8 + liblz4-dev,

9 + libjudy-dev,

10 + libssl-dev,

11 + libmnl-dev,

12 + libjson-c-dev,

13 + libcups2-dev,

14 + libipmimonitoring-dev,

15 + libnetfilter-acct-dev,

16 + autogen,

17 + autoconf,

18 + automake,

19 + pkg-config,

20 + curl,

21 + gcc

22 +Section: net

23 +Priority: optional

24 +Maintainer: Costa Tsaousis <costa@tsaousis.gr>

25 +Standards-Version: 3.9.6

26 +Homepage: https://github.com/netdata/netdata/wiki

27 +

28 +Package: netdata

29 +Architecture: any

30 +Depends: adduser,

31 + libcap2-bin (>= 1:2.0),

32 + lsb-base (>= 3.1-23.2),

33 + zlib1g,

34 + libuuid1,

35 + liblz4-1,

36 + libjudydebian1,

37 + openssl,

38 + libmnl0,

39 + libjson-c3,

40 + cups,

41 + freeipmi,

42 + libnetfilter-acct1,

43 + ${misc:Depends},

44 + ${shlibs:Depends}

45 +Description: real-time charts for system monitoring

46 + Netdata is a daemon that collects data in realtime (per second)

47 + and presents a web site to view and analyze them. The presentation

48 + is also real-time and full of interactive charts that precisely

49 + render all collected values.

contrib/debian/install_go.sh new

+96

@@ -0,0 +1,96 @@

1 +#!/usr/bin/env bash

2 +

3 +LIB_DIR="$1"

4 +LIBEXEC_DIR="$2"

5 +

6 +# ############################################################

7 +# Package Go within netdata (TBD: Package it separately)

8 +safe_sha256sum() {

9 + # Within the contexct of the installer, we only use -c option that is common between the two commands

10 + # We will have to reconsider if we start non-common options

11 + if command -v sha256sum >/dev/null 2>&1; then

12 + sha256sum $@

13 + elif command -v shasum >/dev/null 2>&1; then

14 + shasum -a 256 $@

15 + else

16 + fatal "I could not find a suitable checksum binary to use"

17 + fi

18 +}

19 +

20 +download_go() {

21 + url="${1}"

22 + dest="${2}"

23 +

24 + if command -v curl >/dev/null 2>&1; then

25 + curl -sSL --connect-timeout 10 --retry 3 "${url}" > "${dest}"

26 + elif command -v wget >/dev/null 2>&1; then

27 + wget -T 15 -O - "${url}" > "${dest}"

28 + else

29 + echo >&2

30 + echo >&2 "Downloading go.d plugin from '${url}' failed because of missing mandatory packages."

31 + echo >&2 "Either add packages or disable it by issuing '--disable-go' in the installer"

32 + echo >&2

33 + exit 1

34 + fi

35 +}

36 +

37 +install_go() {

38 + # When updating this value, ensure correct checksums in packaging/go.d.checksums

39 + GO_PACKAGE_VERSION="v0.7.0"

40 + ARCH_MAP=(

41 + 'i386::386'

42 + 'i686::386'

43 + 'x86_64::amd64'

44 + 'aarch64::arm64'

45 + 'armv64::arm64'

46 + 'armv6l::arm'

47 + 'armv7l::arm'

48 + 'armv5tel::arm'

49 + )

50 +

51 + if [ -z "${NETDATA_DISABLE_GO+x}" ]; then

52 + echo >&2 "Install go.d.plugin"

53 + ARCH=$(uname -m)

54 + OS=$(uname -s | tr '[:upper:]' '[:lower:]')

55 +

56 + for index in "${ARCH_MAP[@]}" ; do

57 + KEY="${index%%::*}"

58 + VALUE="${index##*::}"

59 + if [ "$KEY" = "$ARCH" ]; then

60 + ARCH="${VALUE}"

61 + break

62 + fi

63 + done

64 + tmp=$(mktemp -d /tmp/netdata-go-XXXXXX)

65 + GO_PACKAGE_BASENAME="go.d.plugin-${GO_PACKAGE_VERSION}.${OS}-${ARCH}"

66 + download_go "https://github.com/netdata/go.d.plugin/releases/download/${GO_PACKAGE_VERSION}/${GO_PACKAGE_BASENAME}" "${tmp}/${GO_PACKAGE_BASENAME}"

67 + download_go "https://github.com/netdata/go.d.plugin/releases/download/${GO_PACKAGE_VERSION}/config.tar.gz" "${tmp}/config.tar.gz"

68 +

69 + if [ ! -f "${tmp}/${GO_PACKAGE_BASENAME}" ] || [ ! -f "${tmp}/config.tar.gz" ] || [ ! -s "${tmp}/config.tar.gz" ] || [ ! -s "${tmp}/${GO_PACKAGE_BASENAME}" ]; then

70 + echo >&2 "Either check the error or consider disabling it by issuing '--disable-go' in the installer"

71 + echo >&2

72 + return 1

73 + fi

74 +

75 + grep "${GO_PACKAGE_BASENAME}\$" "packaging/go.d.checksums" > "${tmp}/sha256sums.txt" 2>/dev/null

76 + grep "config.tar.gz" "packaging/go.d.checksums" >> "${tmp}/sha256sums.txt" 2>/dev/null

77 +

78 + # Checksum validation

79 + if ! (cd "${tmp}" && safe_sha256sum -c "sha256sums.txt"); then

80 +

81 + echo >&2 "go.d plugin checksum validation failure."

82 + echo >&2 "Either check the error or consider disabling it by issuing '--disable-go' in the installer"

83 + echo >&2

84 +

85 + echo "go.d.plugin package files checksum validation failed."

86 + exit 1

87 + fi

88 +

89 + # Install files

90 + tar -xf "${tmp}/config.tar.gz" -C "${LIB_DIR}/conf.d/"

91 + mv "${tmp}/$GO_PACKAGE_BASENAME" "${LIBEXEC_DIR}/plugins.d/go.d.plugin"

92 + fi

93 + return 0

94 +}

95 +

96 +install_go

contrib/debian/netdata.postinst.in

+4 -3

@@ -28,9 +28,10 @@ case "$1" in

28 fi

30 dpkg-statoverride --update --add --force root netdata 0775 /var/lib/netdata/registry

31 - chown -R root:netdata /usr/share/netdata/*

32 - chown -R root:netdata /usr/lib/@DEB_HOST_MULTIARCH@/netdata/plugins.d

33 - setcap cap_dac_read_search,cap_sys_ptrace+ep /usr/lib/@DEB_HOST_MULTIARCH@/netdata/plugins.d/apps.plugin

31 + chown -R root:netdata /usr/share/netdata

32 + chown -R root:netdata /usr/libexec/netdata/plugins.d

33 + chown -R root:netdata /var/lib/netdata/www

34 + setcap cap_dac_read_search,cap_sys_ptrace+ep /usr/libexec/netdata/plugins.d/apps.plugin

36 #PERMS#

37 ;;

contrib/debian/rules

+16 -6

@@ -17,7 +17,9 @@ TOP = $(CURDIR)/debian/netdata

17 #dh $@ --with autoreconf

19 override_dh_auto_configure:

20 - dh_auto_configure -- --with-math --with-webdir=/var/lib/netdata/www

20 + autoreconf -ivf

21 + dh_auto_configure -- --prefix=/usr --sysconfdir=/etc --localstatedir=/var \

22 + --libexecdir=/usr/libexec --with-user=netdata --with-math --with-webdir=/var/lib/netdata/www

24 debian/%.postinst: debian/%.postinst.in

25 sed 's/@DEB_HOST_MULTIARCH@/$(DEB_HOST_MULTIARCH)/g' $< > $@

@@ -34,7 +36,7 @@ override_dh_install: debian/netdata.postinst

36 mkdir -p "$(TOP)/usr/share/netdata"

37 for D in $$(find "$(TOP)/var/lib/netdata/www/" -maxdepth 1 -type d -printf '%f '); do \

38 echo Relocating $$D; \

37 - mv "$(TOP)/var/lib/netdata/www/$$D" "$(TOP)/usr/share/netdata/$$D"; \

39 + mv "$(TOP)/var/lib/netdata/www/$$D" "$(TOP)/usr/share/netdata/www/$$D"; \

40 ln -s "/usr/share/netdata/$$D" "$(TOP)/var/lib/netdata/www/$$D"; \

41 done

@@ -51,6 +53,10 @@ override_dh_install: debian/netdata.postinst

53 done

54 sed -i "/^#PERMS#/d" $(CURDIR)/debian/netdata.postinst

56 + # Install go

57 + #

58 + debian/install_go.sh $(TOP)/usr/lib/$(DEB_HOST_MULTIARCH)/netdata/ $(TOP)/usr/libexec/netdata

59 +

60 override_dh_installdocs:

61 dh_installdocs

@@ -58,10 +64,11 @@ override_dh_installdocs:

64 -name README.md \

65 -not -path './.travis/*' \

66 -not -path './debian/*' \

67 + -not -path './contrib/*' \

68 -exec cp \

62 - --parents \

63 - --target $(TOP)/usr/share/doc/netdata/ \

64 - {} \;

69 + --parents \

70 + --target $(TOP)/usr/share/doc/netdata/ \

71 + {} \;

73 override_dh_fixperms:

74 dh_fixperms

@@ -69,7 +76,10 @@ override_dh_fixperms:

76 # apps.plugin should only be runnable by the netdata user. It will be

77 # given extra capabilities in the postinst script.

78 #

72 - chmod 0754 $(TOP)/usr/lib/$(DEB_HOST_MULTIARCH)/netdata/plugins.d/apps.plugin

79 + chmod 0754 $(TOP)/usr/libexec/netdata/plugins.d/apps.plugin

80 + chmod 0754 $(TOP)/usr/libexec/netdata/plugins.d/freeipmi.plugin

81 + chmod 0754 $(TOP)/usr/libexec/netdata/plugins.d/perf.plugin

82 + chmod 0750 $(TOP)/usr/libexec/netdata/plugins.d/go.d.plugin

84 override_dh_installlogrotate:

85 cp system/netdata.logrotate debian/netdata.logrotate

packaging/installer/README.md

+3 -1

@@ -298,7 +298,9 @@ To apply the changes you made, you have to restart Netdata.

298

299 We provide our own flavour of binary packages for the most common operating systems that comply with .RPM and .DEB packaging formats.

300

301 -We have currently released .RPM versions with version [1.16.0](https://github.com/netdata/netdata/releases/tag/v1.16.0). We are planning to release packages following the .DEB format on one of the following releases. Our current packaging infrastructure provider is [Package Cloud](https://packagecloud.io).

301 +We have currently released packages following the .RPM format with version [1.16.0](https://github.com/netdata/netdata/releases/tag/v1.16.0).

302 +We have planned to release packages following the .DEB format with version [1.17.0](https://github.com/netdata/netdata/releases/tag/v1.17.0).

303 +Early adopters may experiment with our .DEB formatted packages using our nightly releases. Our current packaging infrastructure provider is [Package Cloud](https://packagecloud.io).

304

305 Netdata is committed to support installation of our solution to all operating systems. This is a constant battle for Netdata, as we strive to automate and make things easier for our users. For the operating system support matrix, please visit our [distributions](../../packaging/DISTRIBUTIONS.md) support page.

306