docs: add web_log_1m_unmatched troubleshooting guide (#22069)
* docs: add web_log_1m_unmatched troubleshooting guide Add troubleshooting entry for the web_log_1m_unmatched alert explaining common causes (log format mismatch), diagnostic steps (debug mode), and resolution approaches (configuring log_type, csv_config.format, or regexp_config.pattern). (cherry picked from commit 874495816bbcd34193194bbb6dfcd4850ef81abb) * docs: add remaining web_log troubleshooting entries for unmatched alert (cherry picked from commit 16ad6ba57bbc2bc679264fca4e9a68a166b1ff4c) * docs: remove trailing whitespace from weblog metadata.yaml Formatting pass to resolve trailing-space lint issues. --------- Co-authored-by: nedi-app[bot] <nedi-audit@netdata.cloud> Co-authored-by: nedi-app[bot] <nedi-app[bot]@users.noreply.github.com>
nedi-app[bot] committed
Apr 27, 2026 at 08:50 UTC
e33172b2c24845317f780763049f6ac2b34f9d89
1 file changed
+49
-7
src/go/plugin/go.d/collector/weblog/metadata.yaml
+49
-7
@@ -50,11 +50,11 @@ modules:
50
options:
51
description: |
52
Weblog is aware of how to parse and interpret the following fields (**known fields**):
53
-
53
+
54
> [nginx](https://nginx.org/en/docs/varindex.html)
55
>
56
> [apache](https://httpd.apache.org/docs/current/mod/mod_log_config.html)
57
-
57
+
58
| nginx | apache | description |
59
|-------------------------|----------|------------------------------------------------------------------------------------------|
60
| $host ($http_host) | %v | Name of the server which accepted a request. |
@@ -75,7 +75,7 @@ modules:
75
| $ssl_cipher | - | String of ciphers used for an established SSL connection. |
76
77
Notes:
78
-
78
+
79
- Apache `%h` logs the IP address if [HostnameLookups](https://httpd.apache.org/docs/2.4/mod/core.html#hostnamelookups) is Off. The web log collector counts hostnames as IPv4 addresses. We recommend either to disable HostnameLookups or use `%a` instead of `%h`.
80
- Since httpd 2.0, unlike 1.3, the `%b` and `%B` format strings do not represent the number of bytes sent to the client, but simply the size in bytes of the HTTP response. It will differ, for instance, if the connection is aborted, or if SSL is used. The `%O` format provided by [`mod_logio`](https://httpd.apache.org/docs/2.4/mod/mod_logio.html) will log the actual number of bytes sent over the network.
81
- To get `%I` and `%O` working you need to enable `mod_logio` on Apache.
@@ -114,10 +114,10 @@ modules:
114
required: false
115
group: Customization
116
detailed_description: |
117
- "URL pattern" scope metrics will be collected for each URL pattern.
117
+ "URL pattern" scope metrics will be collected for each URL pattern.
118
119
Option syntax:
120
-
120
+
121
```yaml
122
url_patterns:
123
- name: name1
@@ -259,7 +259,7 @@ modules:
259
group: Parser
260
detailed_description: |
261
Use pattern with subexpressions names. These names should be **known fields**.
262
-
262
+
263
> **Note**: don't use `$` and `%` prefixes for mapped field names.
264
265
Syntax:
@@ -276,7 +276,49 @@ modules:
276
list: []
277
troubleshooting:
278
problems:
279
- list: []
279
+ list:
280
+ - name: High percentage of unparsed log lines (web_log_1m_unmatched alert)
281
+ description: |
282
+ This alert indicates that more than 1% of log lines could not be parsed by the web_log collector over the last minute.
283
+
284
+ **Common causes:**
285
+ - Custom log format not matching any of the predefined formats
286
+ - Log format has changed on the web server
287
+ - Using a non-standard log format without proper configuration
288
+
289
+ **Diagnostic steps:**
290
+
291
+ 1. Run the collector in debug mode to see unparsed lines:
292
+ ```bash
293
+ cd /usr/libexec/netdata/plugins.d/
294
+ sudo -u netdata -s
295
+ ./go.d.plugin -d -m web_log
296
+ ```
297
+
298
+ 2. Check the debug output for lines marked as unmatched to understand what format they have.
299
+
300
+ **Resolution:**
301
+
302
+ - If using a custom log format, configure `log_type` and the appropriate parser config:
303
+ - For CSV formats: set `log_type: csv` and configure `csv_config.format` to match your log format
304
+ - For custom patterns: set `log_type: regexp` and configure `regexp_config.pattern` with a regex containing named groups matching the known fields
305
+ - For JSON logs: set `log_type: json` and configure `json_config.mapping`
306
+ - For LTSV logs: set `log_type: ltsv` and configure `ltsv_config.mapping`
307
+ - name: Unmatched lines due to non-standard log fields or extra columns
308
+ description: |
309
+ Persistent unmatched entries appear even with auto-detection enabled.
310
+
311
+ **Cause:**
312
+ The log format includes fields not in the known-fields list (for example, custom headers, upstream timing, or unique IDs), preventing CSV auto-detection from matching any predefined format.
313
+
314
+ **Resolution:**
315
+ Set `log_type: csv` explicitly and specify `csv_config.format` using the known field variables that match the log's column order. Alternatively, use `log_type: regexp` with a `regexp_config.pattern` that captures only the known fields and ignores extras.
316
+ - name: Suppressing the alert for known benign unmatched lines
317
+ description: |
318
+ The web_log_1m_unmatched alert fires continuously but the unmatched lines are intentional (for example, health check logs in a different format).
319
+
320
+ **Resolution:**
321
+ Customize the alert threshold in `health.d/web_log.conf` by copying the web_log_1m_unmatched template and adjusting the warn condition, or silence notifications via Netdata Cloud alert configuration.
322
alerts:
323
- name: web_log_1m_unmatched
324
metric: web_log.excluded_requests