| 1 | --- |
| 2 | name: query-netdata-agents |
| 3 | description: Query Netdata Agents (parents and children) directly via their HTTP API on port 19999. Includes a bearer-token helper that mints, caches, and transparently refreshes a per-agent bearer from a long-lived Netdata Cloud token, and auto-detects bearer-protected agents. Use when the user asks how to call an agent's REST API or Function directly, query an agent's logs/metrics/alerts directly, mint a bearer token from a cloud token, or work around bearer protection. |
| 4 | --- |
| 5 | |
| 6 | # Query Netdata Agents directly |
| 7 | |
| 8 | This skill teaches end-users (and AI assistants helping them) how to |
| 9 | talk to a Netdata Agent's HTTP API directly, including |
| 10 | bearer-protected agents that require an SSO-issued bearer token. |
| 11 | |
| 12 | It is the sibling of [`query-netdata-cloud`](../query-netdata-cloud/SKILL.md). |
| 13 | The two skills cover different transports for the same underlying |
| 14 | agent API. |
| 15 | |
| 16 | ## Index of guides |
| 17 | |
| 18 | | Domain | Guide | |
| 19 | |---|---| |
| 20 | | Generic Function invocation | [query-functions.md](./query-functions.md) | |
| 21 | | Logs (`systemd-journal`, `windows-events`, `otel-logs`) | [query-logs.md](./query-logs.md) | |
| 22 | | Topology (`topology:snmp`) | [query-topology.md](./query-topology.md) | |
| 23 | | Flows (`flows:netflow`) | [query-flows.md](./query-flows.md) | |
| 24 | | Alerts (v3 paths) | [query-alerts.md](./query-alerts.md) | |
| 25 | | DynCfg (`/api/v3/config`) | [query-dyncfg.md](./query-dyncfg.md) | |
| 26 | | Time-series metrics (`/api/v3/data`) | [query-metrics.md](./query-metrics.md) | |
| 27 | | Node identity, hardware, vnodes | [query-nodes.md](./query-nodes.md) | |
| 28 | | Streaming (parent / child / replication) -- agent-only | [query-streaming.md](./query-streaming.md) | |
| 29 | | **Operational how-tos (live catalog)** | [how-tos/INDEX.md](./how-tos/INDEX.md) | |
| 30 | |
| 31 | |
| 32 | | Transport | Auth | When to use | |
| 33 | |---|---|---| |
| 34 | | Cloud-proxied (sibling skill) | Cloud token | Default. Works for any team member with cloud access. No agent-side bearer needed. | |
| 35 | | Direct-agent (this skill) | Per-agent bearer (UUID, ~24h TTL) | Power users; lower-latency batch fetches; bypasses the Cloud round-trip; required when Cloud is unavailable. | |
| 36 | |
| 37 | For **what** to query (function payloads, body schemas), see the |
| 38 | sibling skill -- the agent and the Cloud proxy expose the same |
| 39 | Function payload shape. |
| 40 | |
| 41 | This skill ships shell scripts at |
| 42 | [`scripts/_lib.sh`](./scripts/_lib.sh) that automate the bearer mint |
| 43 | / cache / refresh / call-function flow. End-users can either use the |
| 44 | scripts as a black box, or read the script source as a reference |
| 45 | implementation. |
| 46 | |
| 47 | --- |
| 48 | |
| 49 | ## Mandatory Requirements (READ FIRST) |
| 50 | |
| 51 | 1. **If you analyze, you author a how-to.** When asked a concrete |
| 52 | question about an agent that isn't already covered by an |
| 53 | existing how-to under [`how-tos/`](./how-tos/), you MUST author |
| 54 | a new how-to and add it to |
| 55 | [`how-tos/INDEX.md`](./how-tos/INDEX.md) BEFORE completing the |
| 56 | task. The catalog is **live** -- the next assistant should not |
| 57 | redo the same analysis. Keep this catalog operator-facing: |
| 58 | recipes here should explain how to fetch or use Agent data. |
| 59 | Developer contract validation for topology producers, schemas, |
| 60 | fixtures, UI adapters, or aggregator handoffs belongs in the |
| 61 | relevant project developer skill, not in this public skill. |
| 62 | 2. **Use the token-safe wrappers.** `agents_query_cloud`, |
| 63 | `agents_query_agent`, `agents_call_function` from |
| 64 | [`scripts/_lib.sh`](./scripts/_lib.sh) handle auth internally |
| 65 | and emit only the response body to stdout. Never write raw |
| 66 | curl with a literal `Authorization: Bearer $TOKEN` or |
| 67 | `X-Netdata-Auth: Bearer <uuid>`. Bearers / cloud tokens / |
| 68 | claim_ids must NEVER reach assistant-captured stdout. |
| 69 | 3. **Provide actionable instructions.** End every recommendation |
| 70 | with a runnable wrapper invocation. |
| 71 | 4. **Never request credentials.** Use env-key placeholders |
| 72 | (`NETDATA_CLOUD_TOKEN`, `AGENT_EVENTS_HOSTNAME`, |
| 73 | `AGENT_EVENTS_NODE_ID`, etc.) -- the user fills `.env` locally. |
| 74 | 5. **Bearer values stay in `.env` and `.local/`.** The bearer |
| 75 | cache file at `<repo>/.local/audits/query-netdata-agents/ |
| 76 | bearers/<machine_guid>.json` is mode 0600 and gitignored. The |
| 77 | internal helper `_agents_resolve_bearer` returns it through a |
| 78 | validated caller-local output variable, never to stdout. |
| 79 | 6. **For bearer-protected agents, default to the Cloud-token |
| 80 | flow** in this skill (it auto-mints + caches the bearer). |
| 81 | |
| 82 | --- |
| 83 | |
| 84 | ## Prerequisites |
| 85 | |
| 86 | - All [SKILL.md prereqs from `query-netdata-cloud`](../query-netdata-cloud/SKILL.md#prerequisites): |
| 87 | cloud token, space ID, room ID, node UUID. |
| 88 | - Network access to the agent on port 19999 (or whatever it binds). |
| 89 | Test with: `curl -sS http://AGENT_HOST:19999/api/v3/info` -- a 200 |
| 90 | with JSON confirms reachability. |
| 91 | - The agent's `claim_id` if you intend to mint a bearer. It's at |
| 92 | `/api/v3/info` -> `.agents[0].cloud.claim_id`, or with shell |
| 93 | access at `<netdata-prefix>/var/lib/netdata/cloud.d/claimed_id`. |
| 94 | For the install-prefix detection rule, see |
| 95 | [`scripts/_lib.sh`](./scripts/_lib.sh). |
| 96 | |
| 97 | `.env` keys consumed (none are added by this skill -- the four |
| 98 | existing `AGENT_EVENTS_*` keys cover the maintainer-facing |
| 99 | agent-events workflow): |
| 100 | |
| 101 | | Key | Role | |
| 102 | |---|---| |
| 103 | | `NETDATA_CLOUD_TOKEN` | Cloud REST token used to mint per-agent bearers | |
| 104 | | `NETDATA_CLOUD_HOSTNAME` | Cloud REST host | |
| 105 | | `AGENT_EVENTS_HOSTNAME` | When working with the agent-events node specifically -- ssh + direct-HTTP host (IP or DNS name). NOT the journal namespace (hardcoded `agent-events`). | |
| 106 | | `AGENT_EVENTS_NODE_ID` | Target node UUID for direct calls | |
| 107 | | `AGENT_EVENTS_MACHINE_GUID` | Bearer cache key (one bearer per machine_guid) | |
| 108 | |
| 109 | --- |
| 110 | |
| 111 | ## Detect bearer protection |
| 112 | |
| 113 | The signal is HTTP `412 Precondition Failed` from the agent for any |
| 114 | authenticated path (e.g. `/host/<uuid>/api/v3/function?...`). The |
| 115 | response body is `You need to be authorized to access this resource`. |
| 116 | |
| 117 | ```bash |
| 118 | # Probe -- 412 means bearer required, 200 means open access |
| 119 | HOST="agent.example.invalid:19999" |
| 120 | NODE="YOUR_NODE_UUID" |
| 121 | |
| 122 | curl -s -o /dev/null -w '%{http_code}\n' -X POST \ |
| 123 | -H 'Content-Type: application/json' \ |
| 124 | "http://$HOST/host/$NODE/api/v3/function?function=systemd-journal" \ |
| 125 | -d '{"info":true}' |
| 126 | ``` |
| 127 | |
| 128 | The unauthenticated `/api/v3/info` endpoint is always reachable |
| 129 | (returns 200 with the agent's identity). Use it to confirm the host |
| 130 | is up before checking auth. |
| 131 | |
| 132 | --- |
| 133 | |
| 134 | ## Mint a per-agent bearer |
| 135 | |
| 136 | **Endpoint:** `GET /api/v2/bearer_get_token` on Netdata Cloud. |
| 137 | |
| 138 | Required query parameters: `node_id`, `machine_guid`, `claim_id`. |
| 139 | Auth: Cloud token in `Authorization: Bearer ...`. |
| 140 | |
| 141 | ```bash |
| 142 | TOKEN="YOUR_API_TOKEN" |
| 143 | NODE_ID="YOUR_NODE_UUID" |
| 144 | MACHINE_GUID="YOUR_MACHINE_GUID" |
| 145 | CLAIM_ID="YOUR_CLAIM_ID" |
| 146 | |
| 147 | curl -sS \ |
| 148 | -H "Authorization: Bearer $TOKEN" \ |
| 149 | "https://app.netdata.cloud/api/v2/bearer_get_token?node_id=$NODE_ID&machine_guid=$MACHINE_GUID&claim_id=$CLAIM_ID" |
| 150 | ``` |
| 151 | |
| 152 | Response body: |
| 153 | |
| 154 | | Field | Description | |
| 155 | |---|---| |
| 156 | | `token` | The 36-char UUID bearer; pass to the agent in `X-Netdata-Auth: Bearer <token>` | |
| 157 | | `expiration` | Numeric. Format may be Unix ms or seconds; treat values > 10^12 as ms | |
| 158 | | `bearer_protection` | `true` if the agent IS bearer-protected; the token still works either way | |
| 159 | | `mg` | Echoed `machine_guid` | |
| 160 | | `status` | Status code | |
| 161 | |
| 162 | Permission gate (Cloud-side): `PermissionSpaceRead` on the target |
| 163 | space; node must be `reachable`. If the agent is `stale`, the call |
| 164 | returns 400. |
| 165 | |
| 166 | --- |
| 167 | |
| 168 | ## Use the bearer to call an agent |
| 169 | |
| 170 | ```bash |
| 171 | HOST="agent.example.invalid:19999" # the agent's bind address |
| 172 | NODE="YOUR_NODE_UUID" # the node UUID (== nd field) |
| 173 | BEARER="MINTED_BEARER_UUID" |
| 174 | |
| 175 | curl -sS -X POST \ |
| 176 | -H "X-Netdata-Auth: Bearer $BEARER" \ |
| 177 | -H 'Content-Type: application/json' \ |
| 178 | "http://$HOST/host/$NODE/api/v3/function?function=systemd-journal" \ |
| 179 | -d '{"info":true,"timeout":30000}' |
| 180 | ``` |
| 181 | |
| 182 | Notes: |
| 183 | |
| 184 | - The header is **`X-Netdata-Auth: Bearer ...`**, NOT |
| 185 | `Authorization: Bearer ...`. The agent rejects the latter for |
| 186 | per-agent bearer auth. |
| 187 | - The agent's HTTP API path mirrors the Cloud-proxied path. For the |
| 188 | Function payload shape (e.g. `systemd-journal` query body), see |
| 189 | the matching guide in |
| 190 | [`query-netdata-cloud`](../query-netdata-cloud/SKILL.md). |
| 191 | |
| 192 | --- |
| 193 | |
| 194 | ## Bearer cache and refresh |
| 195 | |
| 196 | The shipped scripts cache bearers per `machine_guid` under |
| 197 | `<repo>/.local/audits/query-netdata-agents/bearers/<machine_guid>.json` |
| 198 | (gitignored, mode 0600). Each cache entry stores the raw mint |
| 199 | response. |
| 200 | |
| 201 | Refresh policy: the cache is considered expired when |
| 202 | `expiration - now < 3600` (one-hour buffer before actual TTL). |
| 203 | Mirror of the Cloud frontend's policy |
| 204 | (`cloud-frontend/src/domains/nodes/useAgentBearer.js`). |
| 205 | |
| 206 | A failed mint clears the cache entry so the next call re-mints from |
| 207 | scratch. |
| 208 | |
| 209 | --- |
| 210 | |
| 211 | ## Scripts |
| 212 | |
| 213 | The reference implementation lives in |
| 214 | [`scripts/_lib.sh`](./scripts/_lib.sh). It exposes **token-safe |
| 215 | public wrappers** (the assistant never sees the cloud token, |
| 216 | agent bearer, or claim_id on stdout) and a **self-test** that |
| 217 | asserts no token bytes leak. |
| 218 | |
| 219 | ```bash |
| 220 | # In your script: |
| 221 | source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh" |
| 222 | agents_load_env |
| 223 | |
| 224 | # Cloud-side call. NETDATA_CLOUD_TOKEN is read from .env |
| 225 | # internally; stdout is the response body only. |
| 226 | agents_query_cloud GET /api/v2/spaces |
| 227 | |
| 228 | # Direct-agent call. The bearer is minted/cached/refreshed |
| 229 | # internally. stdout is the response body only; stderr shows the |
| 230 | # curl invocation with `<CLOUD_TOKEN>` and `<AGENT_BEARER>` |
| 231 | # masked. |
| 232 | agents_query_agent \ |
| 233 | --node "$AGENT_EVENTS_NODE_ID" \ |
| 234 | --host "$AGENT_EVENTS_HOSTNAME:19999" \ |
| 235 | --machine-guid "$AGENT_EVENTS_MACHINE_GUID" \ |
| 236 | POST '/api/v3/function?function=systemd-journal' '{"info":true}' |
| 237 | |
| 238 | # Convenience: pick transport with --via cloud|agent. |
| 239 | agents_call_function \ |
| 240 | --via cloud \ |
| 241 | --node "$AGENT_EVENTS_NODE_ID" \ |
| 242 | --function systemd-journal |
| 243 | ``` |
| 244 | |
| 245 | ### Public API (assistant-facing) |
| 246 | |
| 247 | | Function | Purpose | |
| 248 | |---|---| |
| 249 | | `agents_load_env` | Source `<repo>/.env`; validate required keys | |
| 250 | | `agents_repo_root` | Locate this repo's checkout root | |
| 251 | | `agents_audit_dir` | Create + return `<repo>/.local/audits/query-netdata-agents/` | |
| 252 | | `agents_netdata_prefix` | Autodetect Netdata install prefix (system / `/opt/netdata` / `/usr/local/netdata`) | |
| 253 | | `agents_query_cloud METHOD PATH [BODY]` | Call any Cloud REST endpoint. Auth is added internally. **Stdout = response body only.** | |
| 254 | | `agents_query_agent --node N --host H --machine-guid M METHOD PATH [BODY]` | Call any direct-agent path. Bearer resolved internally. **Stdout = response body only.** | |
| 255 | | `agents_call_function --via cloud\|agent --node N --function F [--body J]` | Convenience wrapper around the two above | |
| 256 | | `agents_run` / `agents_run_read` | Run curl with masked-token argv echo on stderr (used by the wrappers; rarely needed directly) | |
| 257 | | `agents_selftest_no_token_leak` | Self-test: drives the wrappers with a sentinel token and asserts the sentinel never reaches captured stdout | |
| 258 | |
| 259 | ### Internal helpers (do NOT call directly) |
| 260 | |
| 261 | These start with `_` and operate on token bytes inside their own |
| 262 | scope. They return token data through validated caller-local output |
| 263 | variables (so the assistant never sees them on stdout). Don't |
| 264 | shell-out to them. |
| 265 | |
| 266 | | Internal | Purpose | |
| 267 | |---|---| |
| 268 | | `_agents_resolve_bearer OUTVAR <node> <mg> <host>` | Cache-aware bearer resolution; writes the bearer into `$OUTVAR` | |
| 269 | | `_agents_get_claim_id OUTVAR <host>` | Resolve `claim_id` from `/api/v3/info`; writes to `$OUTVAR` | |
| 270 | | `_agents_mint_bearer_json <node> <mg> <claim>` | One-shot Cloud bearer mint; the caller MUST capture into a local | |
| 271 | | `_agents_log_masked` | Token / bearer redaction for stderr argv echoes | |
| 272 | | `_agents_exp_to_seconds` | Normalize Cloud `expiration` (sec or ms) to seconds | |
| 273 | |
| 274 | --- |
| 275 | |
| 276 | ## Direct-agent vs Cloud-proxied: how `agents_call_function` chooses |
| 277 | |
| 278 | Default is `--via cloud` -- the safe choice for any team member. |
| 279 | |
| 280 | `--via agent` requires: |
| 281 | 1. The agent host is reachable from the workstation on port 19999. |
| 282 | 2. A bearer (auto-minted internally via `_agents_resolve_bearer`). |
| 283 | |
| 284 | Falls back to `--via cloud` if the direct call fails. |
| 285 | |
| 286 | --- |
| 287 | |
| 288 | ## Sensitive data |
| 289 | |
| 290 | - Bearer values appear in script stderr only when masked. |
| 291 | - The cache file at `.local/audits/.../bearers/<machine_guid>.json` |
| 292 | contains the raw bearer; mode 0600. |
| 293 | - Never paste bearer values, claim ids, machine GUIDs, or node UUIDs |
| 294 | into committed files. See |
| 295 | `<repo>/.agents/sow/specs/sensitive-data-discipline.md` for the |
| 296 | full rule. |