| 1 | # Monitor Nginx or Apache web server log files |
| 2 | |
| 3 | Parsing web server log files with Netdata, revealing the volume of redirects, requests and other metrics, can give you a better overview of your infrastructure. |
| 4 | |
| 5 | Too many bad requests? Maybe a recent deployment missed a few small SVG icons. Too many requests? Time to batten down the hatches—it's a DDoS. |
| 6 | |
| 7 | You can use the [LTSV log format](http://ltsv.org/), track TLS and cipher usage, and the whole parser is faster than |
| 8 | ever. In one test on a system with SSD storage, the collector consistently parsed the logs for 200,000 requests in |
| 9 | 200ms, using ~30% of a single core. |
| 10 | |
| 11 | The [web_log](/src/go/plugin/go.d/collector/weblog/README.md) collector is currently compatible |
| 12 | with [Nginx](https://nginx.org/en/) and [Apache](https://httpd.apache.org/). |
| 13 | |
| 14 | This guide will walk you through using the new Go-based web log collector to turn the logs these web servers |
| 15 | constantly write to into real-time insights into your infrastructure. |
| 16 | |
| 17 | ## Set up your web servers |
| 18 | |
| 19 | As with all data sources, Netdata can auto-detect Nginx or Apache servers if you installed them using their standard |
| 20 | installation procedures. |
| 21 | |
| 22 | Almost all web server installations will need _no_ configuration to start collecting metrics. As long as your web server |
| 23 | has readable access log file, you can configure the web log plugin to access and parse it. |
| 24 | |
| 25 | ## Custom configuration of the web log collector |
| 26 | |
| 27 | The web log collector's default configuration comes with a few example jobs that should cover most Linux distributions |
| 28 | and their default locations for log files: |
| 29 | |
| 30 | ```yaml |
| 31 | # [ JOBS ] |
| 32 | jobs: |
| 33 | # NGINX |
| 34 | # debian, arch |
| 35 | - name: nginx |
| 36 | path: /var/log/nginx/access.log |
| 37 | |
| 38 | # gentoo |
| 39 | - name: nginx |
| 40 | path: /var/log/nginx/localhost.access_log |
| 41 | |
| 42 | # APACHE |
| 43 | # debian |
| 44 | - name: apache |
| 45 | path: /var/log/apache2/access.log |
| 46 | |
| 47 | # gentoo |
| 48 | - name: apache |
| 49 | path: /var/log/apache2/access_log |
| 50 | |
| 51 | # arch |
| 52 | - name: apache |
| 53 | path: /var/log/httpd/access_log |
| 54 | |
| 55 | # debian |
| 56 | - name: apache_vhosts |
| 57 | path: /var/log/apache2/other_vhosts_access.log |
| 58 | |
| 59 | # GUNICORN |
| 60 | - name: gunicorn |
| 61 | path: /var/log/gunicorn/access.log |
| 62 | |
| 63 | - name: gunicorn |
| 64 | path: /var/log/gunicorn/gunicorn-access.log |
| 65 | ``` |
| 66 | |
| 67 | However, if your log files weren’t auto-detected, it might be because they’re in a different location. Try the default |
| 68 | `web_log.conf` file. |
| 69 | |
| 70 | ```bash |
| 71 | ./edit-config go.d/web_log.conf |
| 72 | ``` |
| 73 | |
| 74 | To create a new custom configuration, you need to set the `path` parameter to point to your web server's access log |
| 75 | file. You can give it a `name` as well, and set the `log_type` to `auto`. |
| 76 | |
| 77 | ```yaml |
| 78 | jobs: |
| 79 | - name: example |
| 80 | path: /path/to/file.log |
| 81 | log_type: auto |
| 82 | ``` |
| 83 | |
| 84 | Restart Netdata with `sudo systemctl restart netdata`, or the [appropriate method](/docs/netdata-agent/start-stop-restart.md) for your system. Netdata should pick up your web server's access log and begin showing real-time charts! |
| 85 | |
| 86 | ### Custom log formats and fields |
| 87 | |
| 88 | The web log collector is capable of parsing custom Nginx and Apache log formats and presenting them as charts, but we'll leave that topic for a separate guide. |
| 89 | |
| 90 | We do have [extensive documentation](/src/go/plugin/go.d/collector/weblog/README.md) on how to build custom parsing for Nginx and Apache logs. |
| 91 | |
| 92 | ## Tweak web log collector alerts |
| 93 | |
| 94 | Over time, we've created some default alerts for web log monitoring. These alerts are designed to work only when your |
| 95 | web server is receiving more than 120 requests per minute. Otherwise, there's simply not enough data to make conclusions |
| 96 | about what is "too few" or "too many." |
| 97 | |
| 98 | - [web log alerts](https://raw.githubusercontent.com/netdata/netdata/master/src/health/health.d/web_log.conf). |
| 99 | |
| 100 | You can also edit this file directly with `edit-config`: |
| 101 | |
| 102 | ```bash |
| 103 | ./edit-config health.d/weblog.conf |
| 104 | ``` |
| 105 | |
| 106 | For more information about editing the defaults or writing new alert entities, see our |
| 107 | [health monitoring documentation](/src/health/README.md). |