| 1 | # Running Netdata behind Nginx |
| 2 | |
| 3 | ## Intro |
| 4 | |
| 5 | [Nginx](https://nginx.org/en/) is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server used to host websites and applications of all sizes. |
| 6 | |
| 7 | The software is known for its low impact on memory resources, high scalability, and its modular, event-driven architecture, which can offer secure, predictable performance. |
| 8 | |
| 9 | ## Why Nginx |
| 10 | |
| 11 | - By default, Nginx is fast and lightweight out of the box. |
| 12 | |
| 13 | - Nginx is used and useful in cases when you want to access different instances of Netdata from a single server. |
| 14 | |
| 15 | - Password-protect access to Netdata until distributed authentication is implemented via the Netdata Cloud Sign In mechanism. |
| 16 | |
| 17 | - A proxy was necessary to encrypt the communication to Netdata until v1.16.0, which provided TLS (HTTPS) support. |
| 18 | |
| 19 | ## Nginx configuration file |
| 20 | |
| 21 | All Nginx configurations can be found in the `/etc/nginx/` directory. The main configuration file is `/etc/nginx/nginx.conf`. Website or app-specific configurations can be found in the `/etc/nginx/site-available/` directory. |
| 22 | |
| 23 | Configuration options in Nginx are known as directives. Directives are organized into groups known as blocks or contexts. The two terms can be used interchangeably. |
| 24 | |
| 25 | Depending on your installation source, you’ll find an example configuration file at `/etc/nginx/conf.d/default.conf` or `etc/nginx/sites-enabled/default`, in some cases you may have to manually create the `sites-available` and `sites-enabled` directories. |
| 26 | |
| 27 | You can edit the Nginx configuration file with Nano, Vim or any other text editors you’re comfortable with. |
| 28 | |
| 29 | After making changes to the configuration files: |
| 30 | |
| 31 | - Test Nginx configuration with `nginx -t`. |
| 32 | |
| 33 | - Restart Nginx to effect the change with `/etc/init.d/nginx restart` or `service nginx restart`. |
| 34 | |
| 35 | ## Ways to access Netdata via Nginx |
| 36 | |
| 37 | ### As a virtual host |
| 38 | |
| 39 | With this method instead of `SERVER_IP_ADDRESS:19999`, the Netdata dashboard can be accessed via a human-readable URL such as `netdata.example.com` used in the configuration below. |
| 40 | |
| 41 | ```text |
| 42 | upstream backend { |
| 43 | # the Netdata server |
| 44 | server 127.0.0.1:19999; |
| 45 | keepalive 1024; |
| 46 | } |
| 47 | |
| 48 | server { |
| 49 | # nginx listens to this |
| 50 | listen 80; |
| 51 | # uncomment the line if you want nginx to listen on IPv6 address |
| 52 | #listen [::]:80; |
| 53 | |
| 54 | # the virtual host name of this |
| 55 | server_name netdata.example.com; |
| 56 | |
| 57 | location / { |
| 58 | proxy_set_header X-Forwarded-Host $host; |
| 59 | proxy_set_header X-Forwarded-Server $host; |
| 60 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; |
| 61 | proxy_pass http://backend; |
| 62 | proxy_http_version 1.1; |
| 63 | proxy_pass_request_headers on; |
| 64 | proxy_set_header Connection "keep-alive"; |
| 65 | proxy_store off; |
| 66 | } |
| 67 | } |
| 68 | ``` |
| 69 | |
| 70 | ### As a subfolder to an existing virtual host |
| 71 | |
| 72 | This method is recommended when Netdata is to be served from a subfolder (or directory). |
| 73 | In this case, the virtual host `netdata.example.com` already exists and Netdata has to be accessed via `netdata.example.com/netdata/`. |
| 74 | |
| 75 | ```text |
| 76 | upstream netdata { |
| 77 | server 127.0.0.1:19999; |
| 78 | keepalive 64; |
| 79 | } |
| 80 | |
| 81 | server { |
| 82 | listen 80; |
| 83 | # uncomment the line if you want nginx to listen on IPv6 address |
| 84 | #listen [::]:80; |
| 85 | |
| 86 | # the virtual host name of this subfolder should be exposed |
| 87 | #server_name netdata.example.com; |
| 88 | |
| 89 | location = /netdata { |
| 90 | return 301 /netdata/; |
| 91 | } |
| 92 | |
| 93 | location ~ /netdata/(?<ndpath>.*) { |
| 94 | proxy_redirect off; |
| 95 | proxy_set_header Host $host; |
| 96 | |
| 97 | proxy_set_header X-Forwarded-Host $host; |
| 98 | proxy_set_header X-Forwarded-Server $host; |
| 99 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; |
| 100 | proxy_http_version 1.1; |
| 101 | proxy_pass_request_headers on; |
| 102 | proxy_set_header Connection "keep-alive"; |
| 103 | proxy_store off; |
| 104 | proxy_pass http://netdata/$ndpath$is_args$args; |
| 105 | |
| 106 | gzip on; |
| 107 | gzip_proxied any; |
| 108 | gzip_types *; |
| 109 | } |
| 110 | } |
| 111 | ``` |
| 112 | |
| 113 | ### As a subfolder for multiple Netdata servers, via one Nginx |
| 114 | |
| 115 | This is the recommended configuration when one Nginx will be used to manage multiple Netdata servers via subfolders. |
| 116 | |
| 117 | ```text |
| 118 | upstream backend-server1 { |
| 119 | server 10.1.1.103:19999; |
| 120 | keepalive 64; |
| 121 | } |
| 122 | upstream backend-server2 { |
| 123 | server 10.1.1.104:19999; |
| 124 | keepalive 64; |
| 125 | } |
| 126 | |
| 127 | server { |
| 128 | listen 80; |
| 129 | # uncomment the line if you want nginx to listen on IPv6 address |
| 130 | #listen [::]:80; |
| 131 | |
| 132 | # the virtual host name of this subfolder should be exposed |
| 133 | #server_name netdata.example.com; |
| 134 | |
| 135 | location ~ /netdata/(?<behost>.*?)/(?<ndpath>.*) { |
| 136 | proxy_set_header X-Forwarded-Host $host; |
| 137 | proxy_set_header X-Forwarded-Server $host; |
| 138 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; |
| 139 | proxy_http_version 1.1; |
| 140 | proxy_pass_request_headers on; |
| 141 | proxy_set_header Connection "keep-alive"; |
| 142 | proxy_store off; |
| 143 | proxy_pass http://backend-$behost/$ndpath$is_args$args; |
| 144 | |
| 145 | gzip on; |
| 146 | gzip_proxied any; |
| 147 | gzip_types *; |
| 148 | } |
| 149 | |
| 150 | # make sure there is a trailing slash at the browser |
| 151 | # or the URLs will be wrong |
| 152 | location ~ /netdata/(?<behost>.*) { |
| 153 | return 301 /netdata/$behost/; |
| 154 | } |
| 155 | } |
| 156 | ``` |
| 157 | |
| 158 | Of course, you can add as many backend servers as you like. |
| 159 | |
| 160 | Using the above, you access Netdata on the backend servers like this: |
| 161 | |
| 162 | - `http://netdata.example.com/netdata/server1/` to reach `backend-server1` |
| 163 | - `http://netdata.example.com/netdata/server2/` to reach `backend-server2` |
| 164 | |
| 165 | ### Encrypt the communication between Nginx and Netdata |
| 166 | |
| 167 | In case Netdata's web server has been [configured to use TLS](/src/web/server/README.md#examples), it is |
| 168 | necessary to specify inside the Nginx configuration that the final destination is using TLS. To do this, please, append |
| 169 | the following parameters in your `nginx.conf` |
| 170 | |
| 171 | ```text |
| 172 | proxy_set_header X-Forwarded-Proto https; |
| 173 | proxy_pass https://localhost:19999; |
| 174 | ``` |
| 175 | |
| 176 | Optionally, it is also possible to [enable TLS/SSL on Nginx](http://nginx.org/en/docs/http/configuring_https_servers.html), this way the user will encrypt not only the communication between Nginx and Netdata but also between the user and Nginx. |
| 177 | |
| 178 | If Nginx is not configured as described here, you will probably receive the error `SSL_ERROR_RX_RECORD_TOO_LONG`. |
| 179 | |
| 180 | ### Enable authentication |
| 181 | |
| 182 | :::tip Simpler Alternative |
| 183 | |
| 184 | If you use Netdata Cloud, [Bearer Token Protection](/docs/netdata-agent/configuration/secure-your-netdata-agent-with-bearer-token.md) provides authentication with a single setting - no htpasswd files or nginx auth configuration needed. |
| 185 | |
| 186 | ::: |
| 187 | |
| 188 | Create an authentication file to enable basic authentication via Nginx, this secures your Netdata dashboard. |
| 189 | |
| 190 | If you don't have an authentication file, you can use the following command: |
| 191 | |
| 192 | ```sh |
| 193 | printf "yourusername:$(openssl passwd -apr1)" > /etc/nginx/passwords |
| 194 | ``` |
| 195 | |
| 196 | And then enable the authentication inside your server directive: |
| 197 | |
| 198 | ```text |
| 199 | server { |
| 200 | # ... |
| 201 | auth_basic "Protected"; |
| 202 | auth_basic_user_file passwords; |
| 203 | # ... |
| 204 | } |
| 205 | ``` |
| 206 | |
| 207 | ## Limit direct access to Netdata |
| 208 | |
| 209 | If your Nginx is on `localhost`, you can use this to protect your Netdata: |
| 210 | |
| 211 | ```text |
| 212 | [web] |
| 213 | bind to = 127.0.0.1 ::1 |
| 214 | ``` |
| 215 | |
| 216 | You can also use a unix domain socket. This will also provide a faster route between Nginx and Netdata: |
| 217 | |
| 218 | ```text |
| 219 | [web] |
| 220 | bind to = unix:/var/run/netdata/netdata.sock |
| 221 | ``` |
| 222 | |
| 223 | On the Nginx side, use something like this to use the same unix domain socket: |
| 224 | |
| 225 | ```text |
| 226 | upstream backend { |
| 227 | server unix:/var/run/netdata/netdata.sock; |
| 228 | keepalive 64; |
| 229 | } |
| 230 | ``` |
| 231 | |
| 232 | If your Nginx server is not on localhost, you can set: |
| 233 | |
| 234 | ```text |
| 235 | [web] |
| 236 | bind to = * |
| 237 | allow connections from = IP_OF_NGINX_SERVER |
| 238 | ``` |
| 239 | |
| 240 | `allow connections from` accepts [Netdata simple patterns](/src/libnetdata/simple_pattern/README.md) to match against the |
| 241 | connection IP address. |
| 242 | |
| 243 | ## Prevent the double access.log |
| 244 | |
| 245 | Nginx logs accesses and Netdata logs them too. You can prevent Netdata from generating its access log, by setting this in `/etc/netdata/netdata.conf`: |
| 246 | |
| 247 | ```text |
| 248 | [logs] |
| 249 | access = off |
| 250 | ``` |
| 251 | |
| 252 | ## Use gzip compression |
| 253 | |
| 254 | By default, netdata compresses its responses. You can have nginx do that instead, with the following options in the `location /` block: |
| 255 | |
| 256 | ```text |
| 257 | location / { |
| 258 | ... |
| 259 | gzip on; |
| 260 | gzip_proxied any; |
| 261 | gzip_types *; |
| 262 | } |
| 263 | ``` |
| 264 | |
| 265 | To disable Netdata's gzip compression, open `netdata.conf` and in the `[web]` section put: |
| 266 | |
| 267 | ```text |
| 268 | [web] |
| 269 | enable gzip compression = no |
| 270 | ``` |
| 271 | |
| 272 | ## SELinux |
| 273 | |
| 274 | If you get an 502 Bad Gateway error, you might check your Nginx error log: |
| 275 | |
| 276 | ```sh |
| 277 | # cat /var/log/nginx/error.log: |
| 278 | 2016/09/09 12:34:05 [crit] 5731#5731: *1 connect() to 127.0.0.1:19999 failed (13: Permission denied) while connecting to upstream, client: 1.2.3.4, server: netdata.example.com, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:19999/", host: "netdata.example.com" |
| 279 | ``` |
| 280 | |
| 281 | If you see something like the above, chances are high that SELinux prevents nginx from connecting to the backend server. To fix that, use this policy: `setsebool -P httpd_can_network_connect true`. |