@cryptotaxi247 / netdata-1 / commits / 246e32e27

Update README.md

Costa Tsaousis committed Dec 6, 2023 at 13:45 UTC 246e32e270c86cd68d8db48b2b49e8cd25f2bc73
1 file changed +380 -149
collectors/log2journal/README.md
+380 -149
@@ -1,3 +1,4 @@
1 +
2 # log2journal
3
4 `log2journal` and `systemd-cat-native` can be used to convert a structured log file, such as the ones generated by web servers, into `systemd-journal` entries.
@@ -68,12 +69,11 @@ This pipeline ensures a flexible and comprehensive approach to log processing, a
69
70 ## Real-life example
71
71 -We have an nginx server logging in this format:
72 +We have an nginx server logging in this standard combined log format:
73
74 ```bash
74 - log_format access '$remote_addr - $remote_user [$time_local] '
75 + log_format combined '$remote_addr - $remote_user [$time_local] '
76 '"$request" $status $body_bytes_sent '
76 - '$request_length $request_time '
77 '"$http_referer" "$http_user_agent"';
78 ```
79
@@ -84,85 +84,185 @@ My nginx log uses this log format:
84
85 log_format access '$remote_addr - $remote_user [$time_local] '
86 '"$request" $status $body_bytes_sent '
87 - '$request_length $request_time '
87 '"$http_referer" "$http_user_agent"';
88
89 I want to use `log2joural` to convert this log for systemd-journal.
90 `log2journal` accepts a PCRE2 regular expression, using the named groups
91 in the pattern as the journal fields to extract from the logs.
92
94 -Prefix all PCRE2 group names with `NGINX_` and use capital characters only.
95 -
96 -For the $request, use the field `MESSAGE` (without NGINX_ prefix), so that
97 -it will appear in systemd journals as the message of the log.
98 -
99 -Please give me the PCRE2 pattern.
93 +Please give me the PCRE2 pattern to extract all the fields from my nginx
94 +log files.
95 ```
96
97 ChatGPT replies with this:
98
99 ```regexp
105 -^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>[^"]+)" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"
100 + (?x) # Enable PCRE2 extended mode
101 + ^
102 + (?<remote_addr>[^ ]+) \s - \s
103 + (?<remote_user>[^ ]+) \s
104 + \[
105 + (?<time_local>[^\]]+)
106 + \]
107 + \s+ "
108 + (?<request>
109 + (?<request_method>[A-Z]+) \s+
110 + (?<request_uri>[^ ]+) \s+
111 + (?<server_protocol>[^"]+)
112 + )
113 + " \s+
114 + (?<status>\d+) \s+
115 + (?<body_bytes_sent>\d+) \s+
116 + "(?<http_referer>[^"]*)" \s+
117 + "(?<http_user_agent>[^"]*)"
118 +```
119 +
120 +Let's see what the above says:
121 +
122 +1. `(?x)`: enable PCRE2 extended mode. In this mode spaces and newlines in the pattern are ignored. To match a space you have to use `\s`. This mode allows us to split the pattern is multiple lines and add comments to it.
123 +1. `^`: match the beginning of the line
124 +2. `(?<remote_addr[^ ]+)`: match anything up to the first space (`[^ ]+`), and name it `remote_addr`.
125 +3. `\s`: match a space
126 +4. `-`: match a hyphen
127 +5. and so on...
128 +
129 +We edit `nginx.yaml` and add it, like this:
130 +
131 +```yaml
132 +pattern: |
133 + (?x) # Enable PCRE2 extended mode
134 + ^
135 + (?<remote_addr>[^ ]+) \s - \s
136 + (?<remote_user>[^ ]+) \s
137 + \[
138 + (?<time_local>[^\]]+)
139 + \]
140 + \s+ "
141 + (?<request>
142 + (?<request_method>[A-Z]+) \s+
143 + (?<request_uri>[^ ]+) \s+
144 + (?<server_protocol>[^"]+)
145 + )
146 + " \s+
147 + (?<status>\d+) \s+
148 + (?<body_bytes_sent>\d+) \s+
149 + "(?<http_referer>[^"]*)" \s+
150 + "(?<http_user_agent>[^"]*)"
151 ```
152
153 Let's test it with a sample line (instead of `tail`):
154
155 ```bash
111 -# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>[^"]+)" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"'
112 -MESSAGE=GET /index.html HTTP/1.1
156 +# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal -f nginx.yaml
157 +BODY_BYTES_SENT=4172
158 +HTTP_REFERER=-
159 +HTTP_USER_AGENT=Go-http-client/1.1
160 +REMOTE_ADDR=1.2.3.4
161 +REMOTE_USER=-
162 +REQUEST=GET /index.html HTTP/1.1
163 +REQUEST_METHOD=GET
164 +REQUEST_URI=/index.html
165 +SERVER_PROTOCOL=HTTP/1.1
166 +STATUS=200
167 +TIME_LOCAL=19/Nov/2023:00:24:43 +0000
168 +
169 +```
170 +
171 +As you can see, it extracted all the fields and made them capitals, as systemd-journal expects them.
172 +
173 +To make sure the fields are unique for nginx and do not interfere with other applications, we should prefix them with `NGINX_`:
174 +
175 +```yaml
176 +pattern: |
177 + (?x) # Enable PCRE2 extended mode
178 + ^
179 + (?<remote_addr>[^ ]+) \s - \s
180 + (?<remote_user>[^ ]+) \s
181 + \[
182 + (?<time_local>[^\]]+)
183 + \]
184 + \s+ "
185 + (?<request>
186 + (?<request_method>[A-Z]+) \s+
187 + (?<request_uri>[^ ]+) \s+
188 + (?<server_protocol>[^"]+)
189 + )
190 + " \s+
191 + (?<status>\d+) \s+
192 + (?<body_bytes_sent>\d+) \s+
193 + "(?<http_referer>[^"]*)" \s+
194 + "(?<http_user_agent>[^"]*)"
195 +
196 +prefix: 'NGINX_' # <<< we added this
197 +```
198 +
199 +And let's try it:
200 +
201 +```bash
202 +# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 "-" "Go-http-client/1.1"' | log2journal -f nginx.yaml
203 NGINX_BODY_BYTES_SENT=4172
204 NGINX_HTTP_REFERER=-
205 NGINX_HTTP_USER_AGENT=Go-http-client/1.1
206 NGINX_REMOTE_ADDR=1.2.3.4
207 NGINX_REMOTE_USER=-
118 -NGINX_REQUEST_LENGTH=104
119 -NGINX_REQUEST_TIME=0.001
208 +NGINX_REQUEST=GET /index.html HTTP/1.1
209 +NGINX_REQUEST_METHOD=GET
210 +NGINX_REQUEST_URI=/index.html
211 +NGINX_SERVER_PROTOCOL=HTTP/1.1
212 NGINX_STATUS=200
213 NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
214
215 ```
216
125 -As you can see, it extracted all the fields.
126 -
127 -The `MESSAGE` however, has 3 fields by itself: the method, the URL and the procotol version. Let's ask ChatGPT to extract these too:
128 -
217 +Now, all fields start with `NGINX_` but we want `NGINX_REQUEST` to be the `MESSAGE` of the log line, as we will see it by default in `journalctl` and the Netdata dashboard. Let's rename it:
218 +
219 +```yaml
220 +pattern: |
221 + (?x) # Enable PCRE2 extended mode
222 + ^
223 + (?<remote_addr>[^ ]+) \s - \s
224 + (?<remote_user>[^ ]+) \s
225 + \[
226 + (?<time_local>[^\]]+)
227 + \]
228 + \s+ "
229 + (?<request>
230 + (?<request_method>[A-Z]+) \s+
231 + (?<request_uri>[^ ]+) \s+
232 + (?<server_protocol>[^"]+)
233 + )
234 + " \s+
235 + (?<status>\d+) \s+
236 + (?<body_bytes_sent>\d+) \s+
237 + "(?<http_referer>[^"]*)" \s+
238 + "(?<http_user_agent>[^"]*)"
239 +
240 +prefix: 'NGINX_'
241 +
242 +rename: # <<< we added this
243 + - new_key: MESSAGE # <<< we added this
244 + old_key: NGINX_REQUEST # <<< we added this
245 ```
130 -I see that the MESSAGE has 3 key items in it. The request method (GET, POST,
131 -etc), the URL and HTTP protocol version.
246
133 -I want to keep the MESSAGE as it is, with all the information in it, but also
134 -extract the 3 items from it as separate fields.
135 -
136 -Can this be done?
137 -```
138 -
139 -ChatGPT responded with this:
140 -
141 -```regexp
142 -^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"
143 -```
144 -
145 -Let's test this too:
247 +Let's test it:
248
249 ```bash
148 -# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"'
149 -MESSAGE=GET /index.html HTTP/1.1 # <<<<<<<<< MESSAGE
250 +# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 "-" "Go-http-client/1.1"' | log2journal -f nginx.yaml
251 +MESSAGE=GET /index.html HTTP/1.1 # <<< renamed !
252 NGINX_BODY_BYTES_SENT=4172
253 NGINX_HTTP_REFERER=-
254 NGINX_HTTP_USER_AGENT=Go-http-client/1.1
153 -NGINX_HTTP_VERSION=1.1 # <<<<<<<<< VERSION
154 -NGINX_METHOD=GET # <<<<<<<<< METHOD
255 NGINX_REMOTE_ADDR=1.2.3.4
256 NGINX_REMOTE_USER=-
157 -NGINX_REQUEST_LENGTH=104
158 -NGINX_REQUEST_TIME=0.001
257 +NGINX_REQUEST_METHOD=GET
258 +NGINX_REQUEST_URI=/index.html
259 +NGINX_SERVER_PROTOCOL=HTTP/1.1
260 NGINX_STATUS=200
261 NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
161 -NGINX_URL=/index.html # <<<<<<<<< URL
262
263 ```
264
165 -Ideally, we would want the 5xx errors to be red in our `journalctl` output. To achieve that we need to add a PRIORITY field to set the log level. Log priorities are numeric and follow the `syslog` priorities. Checking `/usr/include/sys/syslog.h` we can see these:
265 +Ideally, we would want the 5xx errors to be red in our `journalctl` output and the dashboard. To achieve that we need to add a PRIORITY field to set the log level. Log priorities are numeric and follow the `syslog` priorities. Checking `/usr/include/sys/syslog.h` we can see these:
266
267 ```c
268 #define LOG_EMERG 0 /* system is unusable */
@@ -175,73 +275,207 @@ Ideally, we would want the 5xx errors to be red in our `journalctl` output. To a
275 #define LOG_DEBUG 7 /* debug-level messages */
276 ```
277
178 -Avoid setting priority to 0 (`LOG_EMERG`), because these will be on your terminal (the journal uses `wall` to let you know of such events). A good priority for errors is 3 (red in `journalctl`), or 4 (yellow in `journalctl`).
179 -
180 -To set the PRIORITY field in the output, we can use `NGINX_STATUS` fields. We need a copy of it, which we will alter later.
278 +Avoid setting priority to 0 (`LOG_EMERG`), because these will be on your terminal (the journal uses `wall` to let you know of such events). A good priority for errors is 3 (red), or 4 (yellow).
279 +
280 +To set the PRIORITY field in the output, we can use `NGINX_STATUS`. We will do this in 2 steps: a) inject the priority field as a copy is `NGINX_STATUS` and then b) use a pattern on its value to rewrite it to the priority level we want.
281 +
282 +First, let's inject it:
283 +
284 +```yaml
285 +pattern: |
286 + (?x) # Enable PCRE2 extended mode
287 + ^
288 + (?<remote_addr>[^ ]+) \s - \s
289 + (?<remote_user>[^ ]+) \s
290 + \[
291 + (?<time_local>[^\]]+)
292 + \]
293 + \s+ "
294 + (?<request>
295 + (?<request_method>[A-Z]+) \s+
296 + (?<request_uri>[^ ]+) \s+
297 + (?<server_protocol>[^"]+)
298 + )
299 + " \s+
300 + (?<status>\d+) \s+
301 + (?<body_bytes_sent>\d+) \s+
302 + "(?<http_referer>[^"]*)" \s+
303 + "(?<http_user_agent>[^"]*)"
304 +
305 +prefix: 'NGINX_'
306 +
307 +rename:
308 + - new_key: MESSAGE
309 + old_key: NGINX_REQUEST
310 +
311 +inject: # <<< we added this
312 + - key: PRIORITY # <<< we added this
313 + value: '${NGINX_STATUS}' # <<< we added this
314 +```
315
182 -We can instruct `log2journal` to duplicate `NGINX_STATUS`, like this: `log2journal --inject 'PRIORITY=${NGINX_STATUS}'`. Let's try it:
316 +Let's see what this does:
317
318 ```bash
185 -# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"' --inject 'PRIORITY=${NGINX_STATUS}'
319 +# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 "-" "Go-http-client/1.1"' | log2journal -f nginx.yaml
320 MESSAGE=GET /index.html HTTP/1.1
321 NGINX_BODY_BYTES_SENT=4172
322 NGINX_HTTP_REFERER=-
323 NGINX_HTTP_USER_AGENT=Go-http-client/1.1
190 -NGINX_HTTP_VERSION=1.1
191 -NGINX_METHOD=GET
324 NGINX_REMOTE_ADDR=1.2.3.4
325 NGINX_REMOTE_USER=-
194 -NGINX_REQUEST_LENGTH=104
195 -NGINX_REQUEST_TIME=0.001
326 +NGINX_REQUEST_METHOD=GET
327 +NGINX_REQUEST_URI=/index.html
328 +NGINX_SERVER_PROTOCOL=HTTP/1.1
329 NGINX_STATUS=200
197 -PRIORITY=200 # <<<<<<<<< PRIORITY IS HERE
330 NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
199 -NGINX_URL=/index.html
331 +PRIORITY=200 # <<< PRIORITY added
332 +
333 +```
334
335 +Now we need to rewrite it to the right priority based on its value. We will assign the priority 6 (info) when the status is 1xx, 2xx, 3xx, priority 5 (notice) when status is 4xx, priority 3 (error) when status is 5xx and anything else will go to priority 4 (warning). Let's do it:
336 +
337 +```yaml
338 +pattern: |
339 + (?x) # Enable PCRE2 extended mode
340 + ^
341 + (?<remote_addr>[^ ]+) \s - \s
342 + (?<remote_user>[^ ]+) \s
343 + \[
344 + (?<time_local>[^\]]+)
345 + \]
346 + \s+ "
347 + (?<request>
348 + (?<request_method>[A-Z]+) \s+
349 + (?<request_uri>[^ ]+) \s+
350 + (?<server_protocol>[^"]+)
351 + )
352 + " \s+
353 + (?<status>\d+) \s+
354 + (?<body_bytes_sent>\d+) \s+
355 + "(?<http_referer>[^"]*)" \s+
356 + "(?<http_user_agent>[^"]*)"
357 +
358 +prefix: 'NGINX_'
359 +
360 +rename:
361 + - new_key: MESSAGE
362 + old_key: NGINX_REQUEST
363 +
364 +inject:
365 + - key: PRIORITY
366 + value: '${NGINX_STATUS}'
367 +
368 +rewrite: # <<< we added this
369 + - key: PRIORITY # <<< we added this
370 + match: '^[123]' # <<< we added this
371 + value: 6 # <<< we added this
372 +
373 + - key: PRIORITY # <<< we added this
374 + match: '^4' # <<< we added this
375 + value: 5 # <<< we added this
376 +
377 + - key: PRIORITY # <<< we added this
378 + match: '^5' # <<< we added this
379 + value: 3 # <<< we added this
380 +
381 + - key: PRIORITY # <<< we added this
382 + match: '.*' # <<< we added this
383 + value: 4 # <<< we added this
384 ```
385
203 -Now that we have the `PRIORITY` field equal to the `NGINX_STATUS`, we can use instruct `log2journal` to change it to a valid priority, by appending: `--rewrite 'PRIORITY=/^5/3' --rewrite 'PRIORITY=/.*/6'`. These rewrite commands say to match everything that starts with `5` and replace it with priority `3` (error) and everything else with priority `6` (info). Let's see it:
386 +Rewrite rules are processed in order and the first matching a field, stops by default processing for this field. This is why the last rule, that matches everything does not always change the priority to 4.
387 +
388 +Let's test it:
389
390 ```bash
206 -# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"' --inject 'PRIORITY=${NGINX_STATUS}' --rewrite 'PRIORITY=/^5/3' --rewrite 'PRIORITY=/.*/6'
391 +# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 "-" "Go-http-client/1.1"' | log2journal -f nginx.yaml
392 MESSAGE=GET /index.html HTTP/1.1
393 NGINX_BODY_BYTES_SENT=4172
394 NGINX_HTTP_REFERER=-
395 NGINX_HTTP_USER_AGENT=Go-http-client/1.1
211 -NGINX_HTTP_VERSION=1.1
212 -NGINX_METHOD=GET
396 NGINX_REMOTE_ADDR=1.2.3.4
397 NGINX_REMOTE_USER=-
215 -NGINX_REQUEST_LENGTH=104
216 -NGINX_REQUEST_TIME=0.001
398 +NGINX_REQUEST_METHOD=GET
399 +NGINX_REQUEST_URI=/index.html
400 +NGINX_SERVER_PROTOCOL=HTTP/1.1
401 NGINX_STATUS=200
218 -PRIORITY=6 # <<<<<<<<<< PRIORITY changed to 6
402 NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
220 -NGINX_URL=/index.html
403 +PRIORITY=6 # <<< PRIORITY rewritten here
404
405 ```
406
224 -Similarly, we could duplicate `${NGINX_URL}` to `NGINX_ENDPOINT` and then process it to remove any query string, or replace IDs in the URL path with constant names, thus giving us uniform endpoints independently of the parameters.
407 +Rewrite rules are powerful. You can have named groups in them, like in the main pattern, to extract sub-fields from them, which you can then use in variable substitution. You can use rewrite rules to anonymize the URLs, e.g to remove customer IDs or transaction details from them.
408 +
409 +To complete the example, we can also inject a `SYSLOG_IDENTIFIER`. Generally your journal logs should always have 3 fields: `MESSAGE`, `PRIORITY` and `SYSLOG_IDENTIFIER`. These 3 fields make it a complete entry. Then you can add as many fields as required for your use case.
410 +
411 +```yaml
412 +pattern: |
413 + (?x) # Enable PCRE2 extended mode
414 + ^
415 + (?<remote_addr>[^ ]+) \s - \s
416 + (?<remote_user>[^ ]+) \s
417 + \[
418 + (?<time_local>[^\]]+)
419 + \]
420 + \s+ "
421 + (?<request>
422 + (?<request_method>[A-Z]+) \s+
423 + (?<request_uri>[^ ]+) \s+
424 + (?<server_protocol>[^"]+)
425 + )
426 + " \s+
427 + (?<status>\d+) \s+
428 + (?<body_bytes_sent>\d+) \s+
429 + "(?<http_referer>[^"]*)" \s+
430 + "(?<http_user_agent>[^"]*)"
431 +
432 +prefix: 'NGINX_'
433 +
434 +rename:
435 + - new_key: MESSAGE
436 + old_key: NGINX_REQUEST
437 +
438 +inject:
439 + - key: PRIORITY
440 + value: '${NGINX_STATUS}'
441 + - key: SYSLOG_IDENTIFIER # <<< we added this
442 + value: 'nginx-log' # <<< we added this
443 +
444 +rewrite:
445 + - key: PRIORITY
446 + match: '^[123]'
447 + value: 6
448 +
449 + - key: PRIORITY
450 + match: '^4'
451 + value: 5
452 +
453 + - key: PRIORITY
454 + match: '^5'
455 + value: 3
456 +
457 + - key: PRIORITY
458 + match: '.*'
459 + value: 4
460 +```
461
226 -To complete the example, we can also inject a `SYSLOG_IDENTIFIER` with `log2journal`, using `--inject SYSLOG_IDENTIFIER=nginx-log`, like this:
462 +Let's see it:
463
464 ```bash
229 -# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"' --inject 'PRIORITY=${NGINX_STATUS}' --inject 'SYSLOG_IDENTIFIER=nginx' -rewrite 'PRIORITY=/^5/3' --rewrite 'PRIORITY=/.*/6'
465 +# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 "-" "Go-http-client/1.1"' | log2journal -f nginx.yaml
466 MESSAGE=GET /index.html HTTP/1.1
467 NGINX_BODY_BYTES_SENT=4172
468 NGINX_HTTP_REFERER=-
469 NGINX_HTTP_USER_AGENT=Go-http-client/1.1
234 -NGINX_HTTP_VERSION=1.1
235 -NGINX_METHOD=GET
470 NGINX_REMOTE_ADDR=1.2.3.4
471 NGINX_REMOTE_USER=-
238 -NGINX_REQUEST_LENGTH=104
239 -NGINX_REQUEST_TIME=0.001
472 +NGINX_REQUEST_METHOD=GET
473 +NGINX_REQUEST_URI=/index.html
474 +NGINX_SERVER_PROTOCOL=HTTP/1.1
475 NGINX_STATUS=200
241 -PRIORITY=6
476 NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
243 -NGINX_URL=/index.html
244 -SYSLOG_IDENTIFIER=nginx-log # <<<<<<<<< THIS HAS BEEN ADDED
477 +PRIORITY=6
478 +SYSLOG_IDENTIFIER=nginx-log # <<< SYSLOG_IDENTIFIER added
479
480 ```
481
@@ -249,105 +483,102 @@ Now the message is ready to be sent to a systemd-journal. For this we use `syste
483
484
485 ```bash
252 -# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"' --inject 'PRIORITY=${NGINX_STATUS}' --inject 'SYSLOG_IDENTIFIER=nginx' -rewrite 'PRIORITY=/^5/3' --rewrite 'PRIORITY=/.*/6' | systemd-cat-native
486 +# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 "-" "Go-http-client/1.1"' | log2journal -f nginx.yaml | systemd-cat-native
487 # no output
488
489 # let's find the message
256 -# journalctl -o verbose SYSLOG_IDENTIFIER=nginx
257 -Sun 2023-11-19 04:34:06.583912 EET [s=1eb59e7934984104ab3b61f5d9648057;i=115b6d4;b=7282d89d2e6e4299969a6030302ff3e4;m=69b419673;t=60a783417ac72;x=2cec5dde8bf01ee7]
490 +# journalctl -r -o verbose SYSLOG_IDENTIFIER=nginx-log
491 +Wed 2023-12-06 13:23:07.083299 EET [s=5290f0133f25407aaa1e2c451c0e4756;i=57194;b=0dfa96ecc2094cecaa8ec0efcb93b865;m=b133308867;t=60bd59346a289;x=5c1bdacf2b9c4bbd]
492 PRIORITY=6
493 _UID=0
494 _GID=0
261 - _BOOT_ID=7282d89d2e6e4299969a6030302ff3e4
262 - _MACHINE_ID=6b72c55db4f9411dbbb80b70537bf3a8
263 - _HOSTNAME=costa-xps9500
495 + _CAP_EFFECTIVE=1ffffffffff
496 + _SELINUX_CONTEXT=unconfined
497 + _BOOT_ID=0dfa96ecc2094cecaa8ec0efcb93b865
498 + _MACHINE_ID=355c8eca894d462bbe4c9422caf7a8bb
499 + _HOSTNAME=lab-logtest-src
500 _RUNTIME_SCOPE=system
501 _TRANSPORT=journal
266 - _CAP_EFFECTIVE=1ffffffffff
502 + MESSAGE=GET /index.html HTTP/1.1
503 + NGINX_BODY_BYTES_SENT=4172
504 + NGINX_HTTP_REFERER=-
505 + NGINX_HTTP_USER_AGENT=Go-http-client/1.1
506 + NGINX_REMOTE_ADDR=1.2.3.4
507 + NGINX_REMOTE_USER=-
508 + NGINX_REQUEST_METHOD=GET
509 + NGINX_REQUEST_URI=/index.html
510 + NGINX_SERVER_PROTOCOL=HTTP/1.1
511 + NGINX_STATUS=200
512 + NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
513 + SYSLOG_IDENTIFIER=nginx-log
514 + _PID=114343
515 + _COMM=systemd-cat-nat
516 + _AUDIT_SESSION=253
517 _AUDIT_LOGINUID=1000
268 - _AUDIT_SESSION=1
269 - _SYSTEMD_CGROUP=/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-59780d3d-a3ff-4a82-a6fe-8d17d2261106.scope
518 + _SYSTEMD_CGROUP=/user.slice/user-1000.slice/session-253.scope
519 + _SYSTEMD_SESSION=253
520 _SYSTEMD_OWNER_UID=1000
271 - _SYSTEMD_UNIT=user@1000.service
272 - _SYSTEMD_USER_UNIT=vte-spawn-59780d3d-a3ff-4a82-a6fe-8d17d2261106.scope
521 + _SYSTEMD_UNIT=session-253.scope
522 _SYSTEMD_SLICE=user-1000.slice
274 - _SYSTEMD_USER_SLICE=app-org.gnome.Terminal.slice
275 - _SYSTEMD_INVOCATION_ID=6195d8c4c6654481ac9a30e9a8622ba1
276 - _COMM=systemd-cat-nat
277 - MESSAGE=GET /index.html HTTP/1.1 # <<<<<<<<< CHECK
278 - NGINX_BODY_BYTES_SENT=4172 # <<<<<<<<< CHECK
279 - NGINX_HTTP_REFERER=- # <<<<<<<<< CHECK
280 - NGINX_HTTP_USER_AGENT=Go-http-client/1.1 # <<<<<<<<< CHECK
281 - NGINX_HTTP_VERSION=1.1 # <<<<<<<<< CHECK
282 - NGINX_METHOD=GET # <<<<<<<<< CHECK
283 - NGINX_REMOTE_ADDR=1.2.3.4 # <<<<<<<<< CHECK
284 - NGINX_REMOTE_USER=- # <<<<<<<<< CHECK
285 - NGINX_REQUEST_LENGTH=104 # <<<<<<<<< CHECK
286 - NGINX_REQUEST_TIME=0.001 # <<<<<<<<< CHECK
287 - NGINX_STATUS=200 # <<<<<<<<< CHECK
288 - NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000 # <<<<<<<<< CHECK
289 - NGINX_URL=/index.html # <<<<<<<<< CHECK
290 - SYSLOG_IDENTIFIER=nginx-log # <<<<<<<<< CHECK
291 - _PID=354312
292 - _SOURCE_REALTIME_TIMESTAMP=1700361246583912
523 + _SYSTEMD_USER_SLICE=-.slice
524 + _SYSTEMD_INVOCATION_ID=c59e33ead8c24880b027e317b89f9f76
525 + _SOURCE_REALTIME_TIMESTAMP=1701861787083299
526 +
527 +```
528 +
529 +So, the log line, with all its fields parsed, ended up in systemd-journal. Now we can send all the nginx logs to systemd-journal like this:
530 +
531 +```bash
532 +tail -F /var/log/nginx/access.log |\
533 + log2journal -f nginx.yaml |\
534 + systemd-cat-native
535 +```
536 +
537 +## Best practices
538 +
539 +**Create a systemd service unit**: Add the above commands to a systemd unit file. When you run it in a systemd unit file you will be able to start/stop it and also see its status. Furthermore you can use the `LogNamespace=` directive of systemd service units to isolate your nginx logs from the logs of the rest of the system. Here is how to do it:
540 +
541 +Create the file `/etc/systemd/system/nginx-logs.service` (change `/path/to/nginx.yaml` to the right path):
542
543 +```
544 +[Unit]
545 +Description=NGINX Log to Systemd Journal
546 +After=network.target
547 +
548 +[Service]
549 +ExecStart=/bin/sh -c 'tail -F /var/log/nginx/access.log | log2journal -f /path/to/nginx.yaml' | systemd-cat-native
550 +LogNamespace=nginx-logs
551 +Restart=always
552 +RestartSec=3
553 +
554 +[Install]
555 +WantedBy=multi-user.target
556 ```
557
296 -So, the log line, with all its fields parsed, ended up in systemd-journal.
558 +Reload systemd to grab this file:
559
298 -The complete example, would look like the following script.
299 -Running this script with parameter `test` will produce output on the terminal for you to inspect.
300 -Unmatched log entries are added to the journal with PRIORITY=1 (`ERR_ALERT`), so that you can spot them.
560 +```bash
561 +sudo systemctl daemon-reload
562 +```
563
302 -We also used the `--filename-key` of `log2journal`, which parses the filename when `tail` switches output
303 -between files, and adds the field `NGINX_LOG_FILE` with the filename each log line comes from.
564 +Enable and start the service:
565
305 -Finally, the script also adds the field `NGINX_STATUS_FAMILY` taking values `2xx`, `3xx`, etc, so that
306 -it is easy to find all the logs of a specific status family.
566 +```bash
567 +sudo systemctl enable nginx-logs.service
568 +sudo systemctl start nginx-logs.service
569 +```
570 +
571 +To see the logs of the namespace, use:
572
573 ```bash
309 -#!/usr/bin/env bash
310 -
311 -test=0
312 -last=0
313 -send_or_show='./systemd-cat-native'
314 -[ "${1}" = "test" ] && test=1 && last=100 && send_or_show=cat
315 -
316 -pattern='(?x) # Enable PCRE2 extended mode
317 -^
318 -(?<NGINX_REMOTE_ADDR>[^ ]+) \s - \s # NGINX_REMOTE_ADDR
319 -(?<NGINX_REMOTE_USER>[^ ]+) \s # NGINX_REMOTE_USER
320 -\[
321 - (?<NGINX_TIME_LOCAL>[^\]]+) # NGINX_TIME_LOCAL
322 -\]
323 -\s+ "
324 -(?<MESSAGE> # MESSAGE
325 - (?<NGINX_METHOD>[A-Z]+) \s+ # NGINX_METHOD
326 - (?<NGINX_URL>[^ ]+) \s+ # NGINX_URL
327 - HTTP/(?<NGINX_HTTP_VERSION>[^"]+) # NGINX_HTTP_VERSION
328 -)
329 -" \s+
330 -(?<NGINX_STATUS>\d+) \s+ # NGINX_STATUS
331 -(?<NGINX_BODY_BYTES_SENT>\d+) \s+ # NGINX_BODY_BYTES_SENT
332 -"(?<NGINX_HTTP_REFERER>[^"]*)" \s+ # NGINX_HTTP_REFERER
333 -"(?<NGINX_HTTP_USER_AGENT>[^"]*)" # NGINX_HTTP_USER_AGENT
334 -'
335 -
336 -tail -n $last -F /var/log/nginx/*access.log \
337 - | log2journal "${pattern}" \
338 - --filename-key 'NGINX_LOG_FILE' \
339 - --unmatched-key 'MESSAGE' \
340 - --inject-unmatched 'PRIORITY=1' \
341 - --inject 'PRIORITY=${NGINX_STATUS}' \
342 - --rewrite 'PRIORITY=/^5/3' \
343 - --rewrite 'PRIORITY=/.*/6' \
344 - --inject 'NGINX_STATUS_FAMILY=${NGINX_STATUS}' \
345 - --rewrite 'NGINX_STATUS_FAMILY=/^(?<first_digit>[0-9]).*$/${first_digit}xx' \
346 - --rewrite 'NGINX_STATUS_FAMILY=/^.*$/UNKNOWN' \
347 - --inject 'SYSLOG_IDENTIFIER=nginx-log' \
348 - | $send_or_show
574 +journalctl -f --namespace=nginx-logs
575 ```
576
577 +Netdata will automatically pick the new namespace and present it at the list of sources of the dashboard.
578 +
579 +You can also instruct `systemd-cat-native` to log to a remote system, sending the logs to a `systemd-journal-remote` instance running on another server. Check [the manual of systemd-cat-native](https://github.com/netdata/netdata/blob/master/libnetdata/log/systemd-cat-native.md).
580 +
581 +
582 ## `log2journal` options
583
584 ```