| 1 | # Query log Functions via Netdata Cloud |
| 2 | |
| 3 | This guide is part of the [`query-netdata-cloud`](./SKILL.md) skill. |
| 4 | Read the [SKILL.md prerequisites](./SKILL.md#prerequisites) first. |
| 5 | For the generic Function transport and the canonical protocol |
| 6 | reference, see [query-functions.md](./query-functions.md). The |
| 7 | authoritative protocol spec is |
| 8 | `<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md` (specifically the |
| 9 | "Log Explorer Format" section). |
| 10 | |
| 11 | Log Functions are the **Log Explorer** class of Functions |
| 12 | (`has_history: true` in their `info` response). They return a |
| 13 | **time-windowed skim** of a larger log dataset, with **facets** |
| 14 | (per-field value counts) for drill-down and an optional |
| 15 | **histogram** (bucketed counts over time) for context. |
| 16 | |
| 17 | Three log Functions exist today, each backed by a different log |
| 18 | source. Their request and response shapes follow the same standard |
| 19 | envelope, but the journal-field set differs per source: |
| 20 | |
| 21 | | Function | Source | Notes | |
| 22 | |---|---|---| |
| 23 | | `systemd-journal` | systemd journal namespaces (system, user, namespace-specific, remote-forwarded) | Linux nodes | |
| 24 | | `windows-events` | Windows event log channels | Windows nodes | |
| 25 | | `otel-logs` | OpenTelemetry logs ingested by the agent | Any node with the OTEL log receiver enabled | |
| 26 | |
| 27 | Confirm which are registered on a node via the |
| 28 | function-listing endpoint in |
| 29 | [query-functions.md](./query-functions.md). Field names below are |
| 30 | illustrative for `systemd-journal`; the same Function payload keys |
| 31 | (`after`, `before`, `last`, `query`, `facets`, `histogram`, |
| 32 | `__logs_sources`, ...) apply to the other two -- only the **values |
| 33 | and column names** differ per source. |
| 34 | |
| 35 | --- |
| 36 | |
| 37 | ## Endpoint |
| 38 | |
| 39 | `POST /api/v2/nodes/{nodeId}/function?function=systemd-journal` |
| 40 | |
| 41 | Same shape as any other Function call. The body is the |
| 42 | `systemd-journal` Function's payload. |
| 43 | |
| 44 | --- |
| 45 | |
| 46 | ## Discover the Function's parameters |
| 47 | |
| 48 | Always start with `info=true` to confirm the current schema -- the |
| 49 | Function's parameter set evolves across agent versions. |
| 50 | |
| 51 | ```bash |
| 52 | TOKEN="YOUR_API_TOKEN" |
| 53 | NODE="YOUR_NODE_UUID" |
| 54 | |
| 55 | read -r -d '' PAYLOAD <<'EOF' |
| 56 | { "info": true } |
| 57 | EOF |
| 58 | |
| 59 | curl -sS -X POST \ |
| 60 | -H 'Content-Type: application/json' \ |
| 61 | -H "Authorization: Bearer $TOKEN" \ |
| 62 | "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=systemd-journal" \ |
| 63 | -d "$PAYLOAD" |
| 64 | ``` |
| 65 | |
| 66 | The `accepted_params` array in the response tells you which keys the |
| 67 | agent currently accepts. |
| 68 | |
| 69 | --- |
| 70 | |
| 71 | ## Body keys (current as of `STATUS_FILE_VERSION = 28`) |
| 72 | |
| 73 | | Key | Type | Purpose | |
| 74 | |---|---|---| |
| 75 | | `info` | bool | Discovery only; do not combine with a real query | |
| 76 | | `after` | int | Unix ms timestamp; lower bound. Negative = relative seconds from `before` | |
| 77 | | `before` | int | Unix ms timestamp; upper bound. Negative = relative seconds from now | |
| 78 | | `last` | int | Page size (rows). Default 200 | |
| 79 | | `direction` | string | `backward` (default; newest first) or `forward` | |
| 80 | | `anchor` | int | Per-row cursor for pagination | |
| 81 | | `query` | string | Free-text search across journal fields | |
| 82 | | `facets` | string[] | Field names to group by (returns counts per value) | |
| 83 | | `histogram` | string | Field name to bucket-by-time | |
| 84 | | `__logs_sources` | string | Source selector. Common values: `all`, `all-local-logs`, `all-local-system-logs`, `all-local-user-logs`, `all-local-namespaces`, plus per-namespace strings like `<namespace-name>` for a specific journal namespace | |
| 85 | | `if_modified_since` | int | Tail mode -- skip if no new data | |
| 86 | | `data_only` | bool | Skip metadata for a faster query | |
| 87 | | `sampling` | int | Cap on rows scanned when search would otherwise be huge | |
| 88 | | `slice` | bool | Native backend filter (faster, less flexible) | |
| 89 | | `delta` | bool | Incremental histogram updates | |
| 90 | | `tail` | bool | Append-mode (combine with `if_modified_since`) | |
| 91 | |
| 92 | `info=true` returns the current authoritative list; rely on it, not |
| 93 | this table, when in doubt. |
| 94 | |
| 95 | --- |
| 96 | |
| 97 | ## Response shape |
| 98 | |
| 99 | The response uses the **standard Function envelope** (top-level |
| 100 | keys `status`, `v`, `type`, `help`, `accepted_params`, |
| 101 | `required_params`, `has_history`, `update_every`, `data`, ...). |
| 102 | For log Functions, `type` is the source name (`logs` family |
| 103 | discriminator). Verified live against the agent-events node: |
| 104 | |
| 105 | | Top-level key | Description | |
| 106 | |---|---| |
| 107 | | `status` | HTTP-style status integer (200 on success) | |
| 108 | | `v` | Function schema version | |
| 109 | | `type` | Family discriminator (carries `logs`-family value) | |
| 110 | | `help` / `accepted_params` / `required_params` | Discovery metadata (see [query-functions.md](./query-functions.md#info-true-discovery)) | |
| 111 | | `data` | **Array** of row arrays -- this is the result rows | |
| 112 | | `columns` | Object keyed by column name; per-column metadata: `index` (position in each row of `data`), `name` (display label), `type` (string / timestamp / integer / ...), `visible`, `unique_key`, `sort`, `summary` (`count` / `min` / `max` / `sum` / ...), `filter` (e.g. `range`), `visualization`, `value_options` (for transforms like `datetime_usec`) | |
| 113 | | `facets` | Array of facet records: `{id, name, options[]}` where each option is `{id, name, count}`. Use to drill down by field value. | |
| 114 | | `histogram` | If requested: time-bucketed counts. Object with `chart`, `id`, `name`, plus per-bucket data | |
| 115 | | `pagination` | Cursor info (`anchor`, `direction`, `last`, ...) for the next page | |
| 116 | | `default_charts` | Suggested chart configuration | |
| 117 | | `default_sort_column` | Recommended sort column | |
| 118 | | `available_histograms` | Field names that the agent can histogram-bucket | |
| 119 | | `_request` | Echo of the parsed request (defaults applied) | |
| 120 | | `versions` | Source/version map for cache invalidation | |
| 121 | | `last_modified` | Last-data timestamp | |
| 122 | | `expires` | Suggested cache expiry | |
| 123 | | `partial` | True if the result was capped by `sampling` or timeout | |
| 124 | | `message` | Optional info / warning string | |
| 125 | | `_journal_files` / `_fstat_caching` / `_sampling` / `_stats` | systemd-journal-specific debug counters | |
| 126 | |
| 127 | ### Reading rows |
| 128 | |
| 129 | `data` is an array of rows. Each row is itself an array whose |
| 130 | positions match `columns.<key>.index`. To pretty-print a single |
| 131 | row by column name: |
| 132 | |
| 133 | ```bash |
| 134 | jq '.columns as $c |
| 135 | | .data[0] as $row |
| 136 | | $c | to_entries |
| 137 | | sort_by(.value.index) |
| 138 | | map({(.key): $row[.value.index]}) |
| 139 | | add' response.json |
| 140 | ``` |
| 141 | |
| 142 | --- |
| 143 | |
| 144 | ## Multi-value field selections (AND-of-OR filtering) |
| 145 | |
| 146 | The `selections` POST-payload key is a structured field-filter |
| 147 | mechanism the Netdata `systemd-journal` Function (powered by the |
| 148 | libnetdata `facets` engine) supports. It is **distinct from raw |
| 149 | journalctl's `KEY=value` matches**: a single field can carry |
| 150 | multiple allowed values, and multiple fields are AND'd. |
| 151 | |
| 152 | ### Shape |
| 153 | |
| 154 | `selections` is an object whose keys are journal field names |
| 155 | and whose values are arrays of allowed values: |
| 156 | |
| 157 | ```json |
| 158 | { |
| 159 | "selections": { |
| 160 | "FIELD1": ["A", "B", "C"], |
| 161 | "FIELD2": ["D", "E"] |
| 162 | } |
| 163 | } |
| 164 | ``` |
| 165 | |
| 166 | Semantics (verified at |
| 167 | `<repo>/src/libnetdata/facets/logs_query_status.h:386-466`): |
| 168 | |
| 169 | - **Between fields: AND.** All listed fields must match. |
| 170 | - **Between values for the same field: OR.** Any one of the |
| 171 | listed values matches. |
| 172 | |
| 173 | So the example above is logically: |
| 174 | |
| 175 | ``` |
| 176 | (FIELD1 in A, B, C) AND (FIELD2 in D, E) |
| 177 | ``` |
| 178 | |
| 179 | ### Why this matters for performance |
| 180 | |
| 181 | A namespace can hold tens of thousands to hundreds of thousands |
| 182 | of records per day. A bare `query` (FTS) scans every record's |
| 183 | indexed text fields. Structured `selections` matches use the |
| 184 | facet engine's per-field index, which is dramatically faster |
| 185 | once the time window is fixed. |
| 186 | |
| 187 | **Rule of thumb:** narrow with `selections` first, then refine |
| 188 | with `query` (FTS) only as a residual narrower over the |
| 189 | already-sliced subset. |
| 190 | |
| 191 | ### Reserved keys inside `selections` |
| 192 | |
| 193 | - `__logs_sources` (per `LQS_PARAMETER_SOURCE`, |
| 194 | `logs_query_status.h:407`) is treated as the source-type |
| 195 | filter (e.g. `all-local-namespaces`, `<namespace-name>`). |
| 196 | Using it inside `selections` is equivalent to setting the |
| 197 | top-level `__logs_sources` parameter. |
| 198 | - `query` inside `selections` is ignored |
| 199 | (`logs_query_status.h:398`); use the top-level `query`. |
| 200 | |
| 201 | ### Example: structured filter + FTS narrower |
| 202 | |
| 203 | ```json |
| 204 | { |
| 205 | "after": -86400, |
| 206 | "before": 0, |
| 207 | "last": 500, |
| 208 | "__logs_sources": "agent-events", |
| 209 | "selections": { |
| 210 | "AE_AGENT_HEALTH": ["crash-first", "crash-loop", "crash-repeated", "crash-entered"], |
| 211 | "AE_AGENT_VERSION": ["v2.10.0", "v2.10.0-135-nightly"] |
| 212 | }, |
| 213 | "query": "deadlock" |
| 214 | } |
| 215 | ``` |
| 216 | |
| 217 | This selects the cross-product of crash-class records on those |
| 218 | two versions (index-resolved), then FTS-filters the result for |
| 219 | the substring `deadlock`. Index-friendly even on a |
| 220 | ~200k-records-per-day namespace. |
| 221 | |
| 222 | ### Anti-pattern (avoid) |
| 223 | |
| 224 | ```json |
| 225 | { |
| 226 | "after": -604800, |
| 227 | "before": 0, |
| 228 | "__logs_sources": "agent-events", |
| 229 | "query": "SIGSEGV" |
| 230 | } |
| 231 | ``` |
| 232 | |
| 233 | A 7-day FTS over the entire namespace with no structured |
| 234 | narrowing. Slow and costly on large namespaces. Always pair FTS |
| 235 | with at least one structured `selections` field. |
| 236 | |
| 237 | --- |
| 238 | |
| 239 | ## Examples |
| 240 | |
| 241 | ### Example 1: most recent 50 entries from a specific namespace |
| 242 | |
| 243 | ```bash |
| 244 | TOKEN="YOUR_API_TOKEN" |
| 245 | NODE="YOUR_NODE_UUID" |
| 246 | NAMESPACE="systemd" # or "agent-events", "any-namespace-name" |
| 247 | |
| 248 | read -r -d '' PAYLOAD <<EOF |
| 249 | { |
| 250 | "after": -3600, |
| 251 | "before": 0, |
| 252 | "last": 50, |
| 253 | "direction": "backward", |
| 254 | "__logs_sources": "${NAMESPACE}" |
| 255 | } |
| 256 | EOF |
| 257 | |
| 258 | curl -sS -X POST \ |
| 259 | -H 'Content-Type: application/json' \ |
| 260 | -H "Authorization: Bearer $TOKEN" \ |
| 261 | "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=systemd-journal" \ |
| 262 | -d "$PAYLOAD" |
| 263 | ``` |
| 264 | |
| 265 | ### Example 2: full-text search with histogram |
| 266 | |
| 267 | ```bash |
| 268 | TOKEN="YOUR_API_TOKEN" |
| 269 | NODE="YOUR_NODE_UUID" |
| 270 | |
| 271 | read -r -d '' PAYLOAD <<'EOF' |
| 272 | { |
| 273 | "after": -86400, |
| 274 | "before": 0, |
| 275 | "last": 100, |
| 276 | "query": "OOM", |
| 277 | "histogram": "PRIORITY", |
| 278 | "facets": ["_SYSTEMD_UNIT", "PRIORITY"], |
| 279 | "__logs_sources": "all-local-system-logs" |
| 280 | } |
| 281 | EOF |
| 282 | |
| 283 | curl -sS -X POST \ |
| 284 | -H 'Content-Type: application/json' \ |
| 285 | -H "Authorization: Bearer $TOKEN" \ |
| 286 | "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=systemd-journal" \ |
| 287 | -d "$PAYLOAD" |
| 288 | ``` |
| 289 | |
| 290 | ### Example 3: paginate forward from a known anchor |
| 291 | |
| 292 | ```bash |
| 293 | TOKEN="YOUR_API_TOKEN" |
| 294 | NODE="YOUR_NODE_UUID" |
| 295 | ANCHOR=1700000123456789 # cursor from previous response |
| 296 | |
| 297 | read -r -d '' PAYLOAD <<EOF |
| 298 | { |
| 299 | "anchor": ${ANCHOR}, |
| 300 | "direction": "forward", |
| 301 | "last": 200, |
| 302 | "__logs_sources": "all-local-logs" |
| 303 | } |
| 304 | EOF |
| 305 | |
| 306 | curl -sS -X POST \ |
| 307 | -H 'Content-Type: application/json' \ |
| 308 | -H "Authorization: Bearer $TOKEN" \ |
| 309 | "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=systemd-journal" \ |
| 310 | -d "$PAYLOAD" |
| 311 | ``` |
| 312 | |
| 313 | --- |
| 314 | |
| 315 | ## Limits and gotchas |
| 316 | |
| 317 | - **Time bounds are unix-microseconds**, not seconds, when given as |
| 318 | positive integers. Negative integers are relative seconds (`-3600` |
| 319 | = "one hour ago relative to `before`"). Mixing units is the most |
| 320 | common bug. |
| 321 | - **Default cloud timeout is 120 s**, but very large queries |
| 322 | (thousands of rows over weeks of data) can hit it. Narrow the |
| 323 | window or use `sampling`. |
| 324 | - **`__logs_sources` is required** to scope to a specific journal |
| 325 | namespace. Without it, the query targets all-local-logs which on a |
| 326 | busy host can be hundreds of GB. |
| 327 | - **Permission**: the cloud token must have a role that includes |
| 328 | log-read access (function tags include `logs`). `scope:all` works; |
| 329 | `scope:grafana-plugin` does NOT. |
| 330 | - **Response can be tens of MB** when `facets` include high-cardinality |
| 331 | fields (`MESSAGE_ID`, `_BOOT_ID`, `_PID`). Pick facets carefully. |
| 332 | |
| 333 | --- |
| 334 | |
| 335 | ## Discovering a journal namespace |
| 336 | |
| 337 | If the host runs `journalctl --namespace=<name>`, the same name is |
| 338 | the value of `__logs_sources`. The agent's `info=true` response |
| 339 | enumerates all visible sources under `accepted_params._logs_sources` |
| 340 | or under a `required_params` widget -- inspect that widget's |
| 341 | `options[]` to learn which sources the node actually exposes. |