878c3f5ae · cryptotaxi247/netdata-1

@@ -77,6 +77,8 @@ We have an nginx server logging in this standard combined log format:

77 '"$http_referer" "$http_user_agent"';

78 ```

79

80 +### Extracting fields with a pattern

81 +

82 First, let's find the right pattern for `log2journal`. We ask ChatGPT:

83

84 ```

@@ -170,6 +172,8 @@ TIME_LOCAL=19/Nov/2023:00:24:43 +0000

172

173 As you can see, it extracted all the fields and made them capitals, as systemd-journal expects them.

174

175 +### Prefixing field names

176 +

177 To make sure the fields are unique for nginx and do not interfere with other applications, we should prefix them with `NGINX_`:

178

179 ```yaml

@@ -214,6 +218,8 @@ NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000

218

219 ```

220

221 +### Renaming fields

222 +

223 Now, all fields start with `NGINX_` but we want `NGINX_REQUEST` to be the `MESSAGE` of the log line, as we will see it by default in `journalctl` and the Netdata dashboard. Let's rename it:

224

225 ```yaml

@@ -262,7 +268,11 @@ NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000

268

269 ```

270

265 -Ideally, we would want the 5xx errors to be red in our `journalctl` output and the dashboard. To achieve that we need to add a PRIORITY field to set the log level. Log priorities are numeric and follow the `syslog` priorities. Checking `/usr/include/sys/syslog.h` we can see these:

271 +### Injecting new fields

272 +

273 +To have a complete message in journals we need 3 fields: `MESSAGE`, `PRIORITY` and `SYSLOG_IDENTIFIER`. We have already added `MESSAGE` by renaming `NGINX_REQUEST`. We can also inject a `SYSLOG_IDENTIFIER` and `PRIORITY`.

274 +

275 +Ideally, we would want the 5xx errors to be red in our `journalctl` output and the dashboard. To achieve that we need to set the `PRIORITY` field to the right log level. Log priorities are numeric and follow the `syslog` priorities. Checking `/usr/include/sys/syslog.h` we can see these:

276

277 ```c

278 #define LOG_EMERG 0 /* system is unusable */

@@ -279,7 +289,7 @@ Avoid setting priority to 0 (`LOG_EMERG`), because these will be on your termina

289

290 To set the PRIORITY field in the output, we can use `NGINX_STATUS`. We will do this in 2 steps: a) inject the priority field as a copy is `NGINX_STATUS` and then b) use a pattern on its value to rewrite it to the priority level we want.

291

282 -First, let's inject it:

292 +First, let's inject `SYSLOG_IDENTIFIER` and `PRIORITY`:

293

294 ```yaml

295 pattern: |

@@ -311,6 +321,9 @@ rename:

321 inject: # <<< we added this

322 - key: PRIORITY # <<< we added this

323 value: '${NGINX_STATUS}' # <<< we added this

324 +

325 + - key: SYSLOG_IDENTIFIER # <<< we added this

326 + value: 'nginx-log' # <<< we added this

327 ```

328

329 Let's see what this does:

@@ -328,11 +341,14 @@ NGINX_REQUEST_URI=/index.html

341 NGINX_SERVER_PROTOCOL=HTTP/1.1

342 NGINX_STATUS=200

343 NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000

331 -PRIORITY=200 # <<< PRIORITY added

344 +PRIORITY=200 # <<< PRIORITY added

345 +SYSLOG_IDENTIFIER=nginx-log # <<< SYSLOG_IDENTIFIER added

346

347 ```

348

335 -Now we need to rewrite it to the right priority based on its value. We will assign the priority 6 (info) when the status is 1xx, 2xx, 3xx, priority 5 (notice) when status is 4xx, priority 3 (error) when status is 5xx and anything else will go to priority 4 (warning). Let's do it:

349 +### Rewriting field values

350 +

351 +Now we need to rewrite `PRIORITY` to the right syslog level based on its value (`NGINX_STATUS`). We will assign the priority 6 (info) when the status is 1xx, 2xx, 3xx, priority 5 (notice) when status is 4xx, priority 3 (error) when status is 5xx and anything else will go to priority 4 (warning). Let's do it:

352

353 ```yaml

354 pattern: |

@@ -400,84 +416,14 @@ NGINX_REQUEST_URI=/index.html

416 NGINX_SERVER_PROTOCOL=HTTP/1.1

417 NGINX_STATUS=200

418 NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000

403 -PRIORITY=6 # <<< PRIORITY rewritten here

419 +PRIORITY=6 # <<< PRIORITY rewritten here

420 +SYSLOG_IDENTIFIER=nginx-log

421

422 ```

423

424 Rewrite rules are powerful. You can have named groups in them, like in the main pattern, to extract sub-fields from them, which you can then use in variable substitution. You can use rewrite rules to anonymize the URLs, e.g to remove customer IDs or transaction details from them.

425

409 -To complete the example, we can also inject a `SYSLOG_IDENTIFIER`. Generally your journal logs should always have 3 fields: `MESSAGE`, `PRIORITY` and `SYSLOG_IDENTIFIER`. These 3 fields make it a complete entry. Then you can add as many fields as required for your use case.

410 -

411 -```yaml

412 -pattern: |

413 - (?x) # Enable PCRE2 extended mode

414 - ^

415 - (?<remote_addr>[^ ]+) \s - \s

416 - (?<remote_user>[^ ]+) \s

417 - \[

418 - (?<time_local>[^\]]+)

419 - \]

420 - \s+ "

421 - (?<request>

422 - (?<request_method>[A-Z]+) \s+

423 - (?<request_uri>[^ ]+) \s+

424 - (?<server_protocol>[^"]+)

425 - )

426 - " \s+

427 - (?<status>\d+) \s+

428 - (?<body_bytes_sent>\d+) \s+

429 - "(?<http_referer>[^"]*)" \s+

430 - "(?<http_user_agent>[^"]*)"

431 -

432 -prefix: 'NGINX_'

433 -

434 -rename:

435 - - new_key: MESSAGE

436 - old_key: NGINX_REQUEST

437 -

438 -inject:

439 - - key: PRIORITY

440 - value: '${NGINX_STATUS}'

441 - - key: SYSLOG_IDENTIFIER # <<< we added this

442 - value: 'nginx-log' # <<< we added this

443 -

444 -rewrite:

445 - - key: PRIORITY

446 - match: '^[123]'

447 - value: 6

448 -

449 - - key: PRIORITY

450 - match: '^4'

451 - value: 5

452 -

453 - - key: PRIORITY

454 - match: '^5'

455 - value: 3

456 -

457 - - key: PRIORITY

458 - match: '.*'

459 - value: 4

460 -```

461 -

462 -Let's see it:

463 -

464 -```bash

465 -# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 "-" "Go-http-client/1.1"' | log2journal -f nginx.yaml

466 -MESSAGE=GET /index.html HTTP/1.1

467 -NGINX_BODY_BYTES_SENT=4172

468 -NGINX_HTTP_REFERER=-

469 -NGINX_HTTP_USER_AGENT=Go-http-client/1.1

470 -NGINX_REMOTE_ADDR=1.2.3.4

471 -NGINX_REMOTE_USER=-

472 -NGINX_REQUEST_METHOD=GET

473 -NGINX_REQUEST_URI=/index.html

474 -NGINX_SERVER_PROTOCOL=HTTP/1.1

475 -NGINX_STATUS=200

476 -NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000

477 -PRIORITY=6

478 -SYSLOG_IDENTIFIER=nginx-log # <<< SYSLOG_IDENTIFIER added

479 -

480 -```

426 +### Sending logs to systemd-journal

427

428 Now the message is ready to be sent to a systemd-journal. For this we use `systemd-cat-native`. This command can send such messages to a journal running on the localhost, a local journal namespace, or a `systemd-journal-remote` running on another server. By just appending `| systemd-cat-native` to the command, the message will be sent to the local journal.

429

Update README.md