| 1 | # Query agent logs directly |
| 2 | |
| 3 | This guide is part of the [`query-netdata-agents`](./SKILL.md) skill. |
| 4 | Read [SKILL.md](./SKILL.md#prerequisites) first. |
| 5 | |
| 6 | For the body shape (`after`, `before`, `last`, `query`, `facets`, |
| 7 | `histogram`, `__logs_sources`, `selections`, etc.) and the |
| 8 | response envelope (top-level `data` is an array of row arrays; |
| 9 | `columns` defines positions; `facets` and `histogram` accompany), |
| 10 | see |
| 11 | [../query-netdata-cloud/query-logs.md](../query-netdata-cloud/query-logs.md). |
| 12 | The body and response are identical between Cloud-proxied and |
| 13 | direct-agent calls -- including the multi-value `selections` |
| 14 | field-filter mechanism (AND across fields, OR across values), |
| 15 | which makes index-friendly queries possible on large namespaces. |
| 16 | See the "Multi-value field selections" section in the Cloud doc |
| 17 | for the exact shape and the structured-filters-first rule. |
| 18 | |
| 19 | The agent ships the same three log Functions: |
| 20 | |
| 21 | - `systemd-journal` (Linux nodes) |
| 22 | - `windows-events` (Windows nodes) |
| 23 | - `otel-logs` (when the OTEL log receiver is enabled) |
| 24 | |
| 25 | --- |
| 26 | |
| 27 | ## Endpoint (agent v3) |
| 28 | |
| 29 | `POST /api/v3/function?function=<log-fn>` on the agent. |
| 30 | |
| 31 | ## Use the wrapper |
| 32 | |
| 33 | ```bash |
| 34 | source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh" |
| 35 | agents_load_env |
| 36 | |
| 37 | # Last-hour skim of a specific journal namespace, 50 rows. |
| 38 | agents_query_agent \ |
| 39 | --node "$AGENT_EVENTS_NODE_ID" \ |
| 40 | --host "$AGENT_EVENTS_HOSTNAME:19999" \ |
| 41 | --machine-guid "$AGENT_EVENTS_MACHINE_GUID" \ |
| 42 | POST '/api/v3/function?function=systemd-journal' \ |
| 43 | '{"after":-3600,"before":0,"last":50,"direction":"backward","__logs_sources":"agent-events"}' |
| 44 | ``` |
| 45 | |
| 46 | The wrapper minted/cached the bearer internally; stdout is the |
| 47 | response body only. The bearer never reaches the assistant's |
| 48 | captured output. |
| 49 | |
| 50 | ## Discover the available log sources |
| 51 | |
| 52 | ```bash |
| 53 | agents_query_agent \ |
| 54 | --node "$AGENT_EVENTS_NODE_ID" \ |
| 55 | --host "$AGENT_EVENTS_HOSTNAME:19999" \ |
| 56 | --machine-guid "$AGENT_EVENTS_MACHINE_GUID" \ |
| 57 | POST '/api/v3/function?function=systemd-journal' '{"info":true}' \ |
| 58 | | jq '.required_params[] | select(.id=="__logs_sources") | .options' |
| 59 | ``` |
| 60 | |
| 61 | Reads the `info=true` response and lists the `__logs_sources` |
| 62 | widget options the agent currently exposes. The `name`+`id` of |
| 63 | each option is what you pass back as the `__logs_sources` value. |
| 64 | |
| 65 | ## Limits and gotchas (single-agent-specific) |
| 66 | |
| 67 | - **Single-host only.** The agent answers for itself; for fleet |
| 68 | queries, use the Cloud-side path or aggregate per-agent |
| 69 | responses client-side. |
| 70 | - **Time bounds**: negative values are seconds-relative-to-now. |
| 71 | Positive values are unix-microseconds (NOT seconds, NOT |
| 72 | milliseconds). Mixing units is the most common bug. |
| 73 | - **Slow queries**: large windows + wide facets can take seconds. |
| 74 | Bump `timeout` in the body to 60000 or higher when the default |
| 75 | 10-second cloud-proxy default isn't relevant (the agent itself |
| 76 | honors the body timeout up to its own ceiling). |
| 77 | |
| 78 | ## See also |
| 79 | |
| 80 | - [../query-netdata-cloud/query-logs.md](../query-netdata-cloud/query-logs.md) |
| 81 | -- full body/response shape, examples, response field |
| 82 | reference. |
| 83 | - [query-functions.md](./query-functions.md) -- the generic |
| 84 | Function transport. |
| 85 | - `<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md` -- canonical |
| 86 | Log Explorer Format spec. |