| 1 | # Query agent Functions via Netdata Cloud |
| 2 | |
| 3 | This guide is part of the [`query-netdata-cloud`](./SKILL.md) skill. |
| 4 | Read the [SKILL.md prerequisites](./SKILL.md#prerequisites) first. |
| 5 | |
| 6 | This file documents the **generic** Function transport: the URL, |
| 7 | the standard response envelope, the `info` discovery query, the |
| 8 | four Function families and where each family's data lives in the |
| 9 | response. |
| 10 | |
| 11 | For three of the four families there is a dedicated guide: |
| 12 | |
| 13 | - **Logs** family (table-history with facets+histogram): |
| 14 | [query-logs.md](./query-logs.md) |
| 15 | - **Topology** family (graph: actors+links): |
| 16 | [query-topology.md](./query-topology.md) |
| 17 | - **Flows** family (network-flow records): |
| 18 | [query-flows.md](./query-flows.md) |
| 19 | |
| 20 | The **table-snapshot** family (full dataset in each response) is |
| 21 | covered here. |
| 22 | |
| 23 | For querying agents directly (without going through Cloud) -- which |
| 24 | includes the transparent Cloud-token to agent-bearer mint flow -- |
| 25 | see the sibling skill |
| 26 | [`query-netdata-agents`](../query-netdata-agents/SKILL.md). |
| 27 | |
| 28 | --- |
| 29 | |
| 30 | ## Mandatory Requirements (READ FIRST) |
| 31 | |
| 32 | 1. **Provide actionable instructions.** Every recommendation ends |
| 33 | in a runnable curl command. |
| 34 | 2. **Never request credentials.** Use `YOUR_API_TOKEN` and |
| 35 | `YOUR_NODE_UUID` placeholders. |
| 36 | 3. **Always start with `{"info":true}`** when you don't already |
| 37 | know the parameter set of the target Function. The `info` |
| 38 | response is authoritative -- this skill's tables can be stale |
| 39 | relative to the running agent. |
| 40 | 4. **Function names are case-sensitive** (e.g. `systemd-journal`, |
| 41 | `topology:snmp`, `flows:netflow`). |
| 42 | |
| 43 | --- |
| 44 | |
| 45 | ## Function classes |
| 46 | |
| 47 | The canonical Functions v3 protocol |
| 48 | (`<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md`) formally defines |
| 49 | **two** Function classes, distinguished by the `has_history` flag |
| 50 | in the `info` response: |
| 51 | |
| 52 | | Class | `has_history` | Frontend behavior | Examples | |
| 53 | |---|---|---|---| |
| 54 | | **Simple Table** | `false` | Backend returns the whole current dataset; frontend filters/sorts/searches in-memory | `processes`, `network-connections`, `network-interfaces`, `network-sockets-tracing`, `block-devices`, `mount-points`, `containers-vms`, `systemd-services`, `netdata-streaming`, `netdata-api-calls`, `netdata-metrics-cardinality`, `<db>:top-queries`, `<db>:running-queries`, `<db>:deadlock-info`, `<db>:error-info` | |
| 55 | | **Log Explorer** | `true` | Backend filters / facets / histograms before sending; supports infinite scroll, anchor pagination, delta and PLAY modes | `systemd-journal`, `windows-events`, `otel-logs` | |
| 56 | |
| 57 | Two additional `type` values are used by purpose-built Functions |
| 58 | that build on the same envelope but emit non-tabular `data`: |
| 59 | |
| 60 | | `type` | Response shape | Examples | Guide | |
| 61 | |---|---|---|---| |
| 62 | | `topology` | `data.actors`/`data.links` graph plus compact-schema sections (`data.evidence`, `data.tables`, `data.overlays`) | `topology:network-connections`, `topology:streaming`, `topology:snmp` | [query-topology.md](./query-topology.md) | |
| 63 | | `flows` | `data.flows[]` plus `data.facets` / `data.columns` / `data.stats` over a time window | `flows:netflow` (covers NetFlow / sFlow / IPFIX) | [query-flows.md](./query-flows.md) | |
| 64 | |
| 65 | For full protocol semantics (facet pills, histograms, charts |
| 66 | configuration, anchor/delta/PLAY modes, error handling, edge |
| 67 | cases), the authoritative source is |
| 68 | `<repo>/src/plugins.d/FUNCTION_UI_REFERENCE.md`. This skill |
| 69 | summarizes the surface that matters for a Cloud-side curl client; |
| 70 | the reference covers everything else. |
| 71 | |
| 72 | --- |
| 73 | |
| 74 | ## Standard response envelope |
| 75 | |
| 76 | Every Function -- regardless of family -- wraps its output in this |
| 77 | envelope. Verified live against the agent's `systemd-journal`, |
| 78 | `topology:snmp`, and `flows:netflow` Functions, and against the |
| 79 | agent emit code at |
| 80 | `src/web/api/functions/function-metrics-cardinality.c:26-39,92` |
| 81 | plus per-collector wrappers. |
| 82 | |
| 83 | | Key | Type | Required | Notes | |
| 84 | |---|---|---|---| |
| 85 | | `status` | int | yes | HTTP-style status (200, 400, ...) | |
| 86 | | `v` | int | yes | Function schema version (currently `3` or `4` depending on Function) | |
| 87 | | `type` | string | yes | Family discriminator: `table`, `logs`, `topology`, `flows` (some Functions emit a custom string -- treat unknown values as `table`-like) | |
| 88 | | `help` | string | typical | Human-readable description | |
| 89 | | `accepted_params` | array<string> | typical | Parameter names accepted in the body | |
| 90 | | `required_params` | array<object> | typical | Per-parameter widget descriptors -- see "info=true discovery" below | |
| 91 | | `has_history` | bool | typical | Whether the Function honors `after` / `before` | |
| 92 | | `update_every` | int | typical | Suggested refresh interval in seconds | |
| 93 | | `data` | array OR object | conditional | Family-specific result. **Absent on `info=true` calls and on errors.** Array for `logs` and `table` families; object (with `actors`/`links` or `flows`/`columns`/`stats`) for `topology` and `flows` | |
| 94 | | `columns` | object | logs / table | Column-metadata, keyed by column name. Each entry has `index` (position inside each row of `data`), `name`, `type`, `visible`, `sort`, `summary`, `filter`, ... | |
| 95 | | `facets` | array | logs / flows | Per-field value distribution and option counts | |
| 96 | | `histogram` | object | logs (when requested) | Bucketed counts over time | |
| 97 | | `pagination` | object | logs | `anchor`, `direction`, `last`, etc. | |
| 98 | | `presentation` | object | topology / flows | Visualization metadata for the Cloud UI | |
| 99 | | `expires` / `last_modified` / `partial` / `message` | scalar | optional | Caching, freshness, partial-result diagnostics | |
| 100 | | `versions` | object | optional | Source/version hashes for client cache invalidation | |
| 101 | |
| 102 | `status >= 400` responses follow the same envelope but include an |
| 103 | `errorMessage` / `errorMsgKey` instead of `data`. |
| 104 | |
| 105 | --- |
| 106 | |
| 107 | ## `info=true` discovery |
| 108 | |
| 109 | The single most important call to make before constructing a real |
| 110 | query: pass `{"info": true}` and read `accepted_params` plus |
| 111 | `required_params`. The agent itself is the authoritative source -- |
| 112 | if a parameter exists there, the Function accepts it; if it |
| 113 | doesn't, no other doc matters. |
| 114 | |
| 115 | ```bash |
| 116 | TOKEN="YOUR_API_TOKEN" |
| 117 | NODE="YOUR_NODE_UUID" |
| 118 | FN="systemd-journal" |
| 119 | |
| 120 | read -r -d '' PAYLOAD <<'EOF' |
| 121 | { "info": true } |
| 122 | EOF |
| 123 | |
| 124 | curl -sS -X POST \ |
| 125 | -H 'Content-Type: application/json' \ |
| 126 | -H "Authorization: Bearer $TOKEN" \ |
| 127 | "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=$FN" \ |
| 128 | -d "$PAYLOAD" |
| 129 | ``` |
| 130 | |
| 131 | ### `required_params` widget schema |
| 132 | |
| 133 | Each entry of `required_params` is a UI-widget descriptor that |
| 134 | tells a client what to render and what values are valid. Verified |
| 135 | against the emit code in |
| 136 | `src/collectors/network-viewer.plugin/network-viewer.c:1601-1731` |
| 137 | and across the topology / logs / flows Functions. |
| 138 | |
| 139 | | Field | Type | Required | Purpose | |
| 140 | |---|---|---|---| |
| 141 | | `id` | string | yes | Parameter id (the body key) | |
| 142 | | `name` | string | yes | Display label | |
| 143 | | `help` | string | typical | Tooltip / help text | |
| 144 | | `type` | string | yes | Widget kind -- see table below | |
| 145 | | `options[]` | array | for select/multiselect/autocomplete | Each option: `{ "id": "<value>", "name": "<label>", "defaultSelected": <bool>? }` | |
| 146 | | `unique_view` | bool | optional | Single-select enforces single-value semantics | |
| 147 | | `multiselect` | bool | optional | Multi-value semantics | |
| 148 | | `pattern` | string | optional | Regex/glob input for `text`/`pattern` widgets | |
| 149 | | `default_value` | scalar | optional | Pre-filled value | |
| 150 | |
| 151 | Widget `type` values seen in source: |
| 152 | |
| 153 | | `type` | Meaning | |
| 154 | |---|---| |
| 155 | | `select` | Single-choice dropdown | |
| 156 | | `multiselect` | Multi-choice; each option may have `defaultSelected:true` | |
| 157 | | `autocomplete` | Text input backed by an autocomplete query (the Function itself answers via `mode:"autocomplete"` or similar) | |
| 158 | | `text` | Free-form text | |
| 159 | | `checkbox` | Boolean toggle | |
| 160 | | `range` | Numeric range / slider (newer Functions) | |
| 161 | | `pattern` | Pattern / regex input (newer Functions) | |
| 162 | |
| 163 | The widget array is the contract between the agent and any UI or |
| 164 | script. To programmatically construct a valid body for a Function, |
| 165 | walk `required_params` and emit the body shape it implies. There |
| 166 | is no central widget builder API in source -- each collector emits |
| 167 | the array directly via `buffer_json_*` calls -- so the agent's own |
| 168 | `info=true` response is the only authoritative place to read the |
| 169 | schema for a specific node version. |
| 170 | |
| 171 | --- |
| 172 | |
| 173 | ## Endpoints |
| 174 | |
| 175 | ### List Functions on the nodes in a room |
| 176 | |
| 177 | `POST /api/v3/spaces/{spaceID}/rooms/{roomID}/functions` |
| 178 | |
| 179 | ```bash |
| 180 | TOKEN="YOUR_API_TOKEN" |
| 181 | SPACE="YOUR_SPACE_ID" |
| 182 | ROOM="YOUR_ROOM_ID" |
| 183 | |
| 184 | read -r -d '' PAYLOAD <<'EOF' |
| 185 | { |
| 186 | "scope": { "nodes": [] }, |
| 187 | "selectors": { "nodes": ["*"] } |
| 188 | } |
| 189 | EOF |
| 190 | |
| 191 | curl -sS -X POST \ |
| 192 | -H 'Content-Type: application/json' \ |
| 193 | -H "Authorization: Bearer $TOKEN" \ |
| 194 | "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/functions" \ |
| 195 | -d "$PAYLOAD" |
| 196 | ``` |
| 197 | |
| 198 | Response top-level: `functions[]` (each entry: `name`, `version`, |
| 199 | `help`, `ni[]`, `tags`, `access[]`, `priority`), `nodes[]` (each |
| 200 | `{ ni, mg, nd, nm, st }`), `agents[]`, `versions`. Match |
| 201 | `functions[].ni` to `nodes[].ni` to find which nodes expose a |
| 202 | given Function. |
| 203 | |
| 204 | ### Invoke a Function on a node |
| 205 | |
| 206 | `POST /api/v2/nodes/{nodeId}/function?function={functionName}` |
| 207 | |
| 208 | ```bash |
| 209 | TOKEN="YOUR_API_TOKEN" |
| 210 | NODE="YOUR_NODE_UUID" |
| 211 | FN="processes" |
| 212 | |
| 213 | read -r -d '' PAYLOAD <<'EOF' |
| 214 | { |
| 215 | "last": 50, |
| 216 | "timeout": 30000 |
| 217 | } |
| 218 | EOF |
| 219 | |
| 220 | curl -sS -X POST \ |
| 221 | -H 'Content-Type: application/json' \ |
| 222 | -H "Authorization: Bearer $TOKEN" \ |
| 223 | "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=$FN" \ |
| 224 | -d "$PAYLOAD" |
| 225 | ``` |
| 226 | |
| 227 | Optional headers: |
| 228 | |
| 229 | | Header | Purpose | |
| 230 | |---|---| |
| 231 | | `X-Transaction-Id: <uuid>` | Correlation id propagated to the agent. Optional. | |
| 232 | |
| 233 | --- |
| 234 | |
| 235 | ## Frequently registered Functions |
| 236 | |
| 237 | Function availability is per-node. The listing endpoint above is |
| 238 | the only authoritative source. Below are common Functions on a |
| 239 | stock Linux Netdata install (verified live): |
| 240 | |
| 241 | | Function | Family | What it returns | |
| 242 | |---|---|---| |
| 243 | | `processes` | table | Live process list with CPU / memory / I/O / page faults / PPID | |
| 244 | | `network-connections` | table | Active sockets/connections (proto, state, addresses, ports, perf metrics) | |
| 245 | | `network-interfaces` | table | Per-interface traffic, packet counts, drops, link status | |
| 246 | | `network-sockets-tracing` | table | Detailed open-socket information | |
| 247 | | `block-devices` | table | Per-block-device read/write throughput, ops, latency, utilization | |
| 248 | | `mount-points` | table | Filesystem mount points with space and inode usage | |
| 249 | | `containers-vms` | table | Active containers and cgroups with resource usage | |
| 250 | | `systemd-services` | table | systemd service cgroups with process counts and resource use | |
| 251 | | `netdata-streaming` | table | Parent-child streaming/replication status, data-flow metrics, ML status | |
| 252 | | `netdata-api-calls` | table | Active and recent Netdata API requests with timings | |
| 253 | | `netdata-metrics-cardinality` | table | Cardinality stats (instances, time-series per context/node) | |
| 254 | | `systemd-journal` | logs | systemd journal entries -- see [query-logs.md](./query-logs.md) | |
| 255 | | `windows-events` | logs | Windows event log channels (Windows nodes only) | |
| 256 | | `otel-logs` | logs | OpenTelemetry log entries (when the OTEL log receiver is enabled) | |
| 257 | | `topology:snmp` | topology | LLDP/CDP/FDB/STP-derived L2 topology -- see [query-topology.md](./query-topology.md) | |
| 258 | | `flows:netflow` | flows | NetFlow / sFlow / IPFIX records -- see [query-flows.md](./query-flows.md) | |
| 259 | |
| 260 | Database collectors register a per-collector family of Functions |
| 261 | when active: `<collector>:top-queries`, `<collector>:running-queries`, |
| 262 | `<collector>:deadlock-info`, `<collector>:error-info` -- e.g. |
| 263 | `postgres:top-queries`, `mysql:top-queries`, `mssql:deadlock-info`. |
| 264 | The listing endpoint reports them when the collector is enabled. |
| 265 | |
| 266 | --- |
| 267 | |
| 268 | ## Examples (table-snapshot Functions) |
| 269 | |
| 270 | For logs / topology / flows examples, see the per-family guides |
| 271 | linked at the top. |
| 272 | |
| 273 | ### Example 1: top processes by CPU |
| 274 | |
| 275 | ```bash |
| 276 | TOKEN="YOUR_API_TOKEN" |
| 277 | NODE="YOUR_NODE_UUID" |
| 278 | |
| 279 | read -r -d '' PAYLOAD <<'EOF' |
| 280 | { |
| 281 | "last": 50, |
| 282 | "timeout": 30000 |
| 283 | } |
| 284 | EOF |
| 285 | |
| 286 | curl -sS -X POST \ |
| 287 | -H 'Content-Type: application/json' \ |
| 288 | -H "Authorization: Bearer $TOKEN" \ |
| 289 | "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=processes" \ |
| 290 | -d "$PAYLOAD" \ |
| 291 | | jq '.data | length, (.[0:3])' |
| 292 | ``` |
| 293 | |
| 294 | ### Example 2: discover a Function's parameter widget set |
| 295 | |
| 296 | ```bash |
| 297 | read -r -d '' PAYLOAD <<'EOF' |
| 298 | { "info": true } |
| 299 | EOF |
| 300 | |
| 301 | curl -sS -X POST \ |
| 302 | -H 'Content-Type: application/json' \ |
| 303 | -H "Authorization: Bearer $TOKEN" \ |
| 304 | "https://app.netdata.cloud/api/v2/nodes/$NODE/function?function=network-connections" \ |
| 305 | -d "$PAYLOAD" \ |
| 306 | | jq '.required_params | map({id, type, name, options: (.options | length // 0)})' |
| 307 | ``` |
| 308 | |
| 309 | ### Example 3: list the Functions on a single node |
| 310 | |
| 311 | ```bash |
| 312 | read -r -d '' PAYLOAD <<'EOF' |
| 313 | { |
| 314 | "scope": { "nodes": ["YOUR_NODE_UUID"] }, |
| 315 | "selectors": { "nodes": ["*"] } |
| 316 | } |
| 317 | EOF |
| 318 | |
| 319 | curl -sS -X POST \ |
| 320 | -H 'Content-Type: application/json' \ |
| 321 | -H "Authorization: Bearer $TOKEN" \ |
| 322 | "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/functions" \ |
| 323 | -d "$PAYLOAD" \ |
| 324 | | jq -r '.functions[] | "\(.name)\t\(.tags // "")\t\(.help)"' |
| 325 | ``` |
| 326 | |
| 327 | --- |
| 328 | |
| 329 | ## Limits and gotchas |
| 330 | |
| 331 | - **Cloud default timeout is 120 s** for Function calls; pass |
| 332 | `"timeout": <ms>` in the body for slower Functions but Cloud |
| 333 | may impose its own ceiling. |
| 334 | - **Response is NOT streamed.** The Cloud proxy collects the full |
| 335 | agent response and returns it in one body. For potentially |
| 336 | huge results (logs, flows), narrow the time window or use the |
| 337 | Function's pagination (`last`, `anchor`) rather than relying on |
| 338 | streaming. |
| 339 | - **Node must be `reachable`.** A `stale` node returns HTTP 400 |
| 340 | with `errorMsgKey: "ErrInstanceNotReachable"`. Verify with the |
| 341 | discovery endpoints in [SKILL.md](./SKILL.md). |
| 342 | - **Permission**: the cloud token must include |
| 343 | `PermissionFunctionExec` on the target space. `scope:all` |
| 344 | works; `scope:grafana-plugin` does NOT. |
| 345 | - **Function name is case-sensitive** -- wrong casing returns 400. |
| 346 | - **`info=true` does NOT bypass auth.** ACL is enforced on every |
| 347 | call regardless of body. |
| 348 | - **The agent's own `info=true` response is authoritative for |
| 349 | parameters.** Tables in this skill can drift relative to the |
| 350 | running version. When in doubt, ask the agent. |