Added health check functionality to our Docker images. (#9172)
* Add a `ping` command to netdatacli to check if agent is alive. This provides a way to trivially check if the agent itself appears to be running (namely, the command parser for netdatacli in the agent itself is working and responding), allowing users to check this without having to rely on us continuing to have `help` be a command sent to the agent instead of executing locally. * Add a basic health check to our docke rimages. This adds a relatively basic health checker script to our Docker images. By default it verifies that the `/api/v1/info` endpoint returns a 200 status code. It also supports checking different endpoints or using `netdatacli ping` to check that Netdata is running, all controlled by a new Docker environment variable: `NETDATA_HEALTH_CHECK`. * Avoid unnessecary `chmod` in Dockerfile. Suggested by @prologic. * Fix typo in docs. * Update environment variable name to be more clear. Also add `-L` to `curl` command in health check to follow redirects.
Austin S. Hemmelgarn committed
May 28, 2020 at 04:29 UTC
ed8a6bea174bc4c52d4af4f4b138b5fe98003b49
5 files changed
+59
-2
daemon/commands.c
+15
-2
@@ -45,6 +45,7 @@ static cmd_status_t cmd_reload_claiming_state_execute(char *args, char **message
45
static cmd_status_t cmd_reload_labels_execute(char *args, char **message);
46
static cmd_status_t cmd_read_config_execute(char *args, char **message);
47
static cmd_status_t cmd_write_config_execute(char *args, char **message);
48
+static cmd_status_t cmd_ping_execute(char *args, char **message);
49
50
static command_info_t command_info_array[] = {
51
{"help", cmd_help_execute, CMD_TYPE_HIGH_PRIORITY}, // show help menu
@@ -56,7 +57,8 @@ static command_info_t command_info_array[] = {
57
{"reload-claiming-state", cmd_reload_claiming_state_execute, CMD_TYPE_ORTHOGONAL}, // reload claiming state
58
{"reload-labels", cmd_reload_labels_execute, CMD_TYPE_ORTHOGONAL}, // reload the labels
59
{"read-config", cmd_read_config_execute, CMD_TYPE_CONCURRENT},
59
- {"write-config", cmd_write_config_execute, CMD_TYPE_ORTHOGONAL}
60
+ {"write-config", cmd_write_config_execute, CMD_TYPE_ORTHOGONAL},
61
+ {"ping", cmd_ping_execute, CMD_TYPE_ORTHOGONAL}
62
};
63
64
/* Mutexes for commands of type CMD_TYPE_ORTHOGONAL */
@@ -117,7 +119,9 @@ static cmd_status_t cmd_help_execute(char *args, char **message)
119
"fatal-agent\n"
120
" Log the state and halt the netdata agent.\n"
121
"reload-claiming-state\n"
120
- " Reload agent claiming state from disk.\n",
122
+ " Reload agent claiming state from disk.\n"
123
+ "ping\n"
124
+ " Return with 'pong' if agent is alive.\n",
125
MAX_COMMAND_LENGTH - 1);
126
return CMD_STATUS_SUCCESS;
127
}
@@ -296,6 +300,15 @@ static cmd_status_t cmd_write_config_execute(char *args, char **message)
300
return CMD_STATUS_SUCCESS;
301
}
302
303
+static cmd_status_t cmd_ping_execute(char *args, char **message)
304
+{
305
+ (void)args;
306
+
307
+ *message = strdupz("pong");
308
+
309
+ return CMD_STATUS_SUCCESS;
310
+}
311
+
312
static void cmd_lock_exclusive(unsigned index)
313
{
314
(void)index;
daemon/commands.h
+1
@@ -23,6 +23,7 @@ typedef enum cmd {
23
CMD_RELOAD_LABELS,
24
CMD_READ_CONFIG,
25
CMD_WRITE_CONFIG,
26
+ CMD_PING,
27
CMD_TOTAL_COMMANDS
28
} cmd_t;
29
packaging/docker/Dockerfile
+3
@@ -60,6 +60,7 @@ FROM netdata/base:${ARCH}
60
# Copy files over
61
RUN mkdir -p /opt/src
62
COPY --from=builder /app /
63
+COPY packaging/docker/health.sh /health.sh
64
65
# Configure system
66
ARG NETDATA_UID=201
@@ -106,3 +107,5 @@ ENV NETDATA_PORT 19999
107
EXPOSE $NETDATA_PORT
108
109
ENTRYPOINT ["/usr/sbin/run.sh"]
110
+
111
+HEALTHCHECK --interval=60s --timeout=10s --retries=3 CMD /health.sh
packaging/docker/README.md
+23
@@ -98,6 +98,29 @@ volumes:
98
99
Run `docker-compose up -d` in the same directory as the `docker-compose.yml` file to start the container.
100
101
+## Health Checks
102
+
103
+Our Docker image provides integrated support for health checks through the standard Docker interfaces.
104
+
105
+You can control how the health checks run by using the environment variable `NETDATA_HEALTHCHECK_TARGET` as follows:
106
+
107
+- If left unset, the health check will attempt to access the
108
+ `/api/v1/info` endpoint of the agent.
109
+- If set to the exact value 'cli', the health check
110
+ script will use `netdatacli ping` to determine if the agent is running
111
+ correctly or not. This is sufficient to ensure that Netdata did not
112
+ hang during startup, but does not provide a rigorous verification
113
+ that the daemon is collecting data or is otherwise usable.
114
+- If set to anything else, the health check will treat the vaule as a
115
+ URL to check for a 200 status code on. In most cases, this should
116
+ start with `http://localhost:19999/` to check the agent running in
117
+ the container.
118
+
119
+In most cases, the default behavior of checking the `/api/v1/info`
120
+endpoint will be sufficient. If you are using a configuration which
121
+disables the web server or restricts access to certain API's, you will
122
+need to use a non-default configuration for health checks to work.
123
+
124
## Configure Agent containers
125
126
You may need to configure the above `docker run...` and `docker-compose` commands based on your needs. You should
packaging/docker/health.sh
new
+17
@@ -0,0 +1,17 @@
1
+#!/bin/sh
2
+#
3
+# This is the script that gets run for our Docker image health checks.
4
+
5
+if [ -z "${NETDATA_HEALTHCHECK_TARGET}" ] ; then
6
+ # If users didn't request something else, query `/api/v1/info`.
7
+ NETDATA_HEALTHCHECK_TARGET="http://localhost:19999/api/v1/info"
8
+fi
9
+
10
+case "${NETDATA_HEALTHCHECK_TARGET}" in
11
+ cli)
12
+ netdatacli ping || exit 1
13
+ ;;
14
+ *)
15
+ curl -sSL "${NETDATA_HEALTHCHECK_TARGET}" || exit 1
16
+ ;;
17
+esac