@cryptotaxi247 / netdata-1 / commits / 3902e7fde

Improve query metrics skill document (#21836)

Improve query metrics skill: unaligned default, pa bitmap, scope.contexts, selectors.nodes - Add `unaligned` to all examples and the request template; explain why API queries should always use it (wall-clock alignment confuses users) - Document `pa` point annotations as a bitmap with all 3 flags: empty (1), reset (2), partial (4) - Make `scope.contexts` mandatory: add to Mandatory Requirements, mark as REQUIRED in template and table, add warnings about metadata explosion (verified: 8KB vs 2.5MB without it) - Prefer `selectors.nodes` (hostname patterns) over `scope.nodes` (UUIDs) for node filtering simplicity - Add bottom reminder for scope.contexts and unaligned - Reorder Known Limitations to lead with scope.contexts

Costa Tsaousis committed Feb 27, 2026 at 02:08 UTC 3902e7fdec7b8d25f21f346e55cd6728c487aeb7

1 file changed +50 -59

docs/netdata-ai/skills/query-netdata-cloud-metrics.md

+50 -59

@@ -1,14 +1,14 @@

1 -# Skill: Query Netdata Cloud Metrics

2 -

3 -Help users query time-series metrics from Netdata Cloud via the REST API.

1 +# Query time-series metrics from Netdata Cloud via the REST API.

3 ## Mandatory Requirements (READ FIRST)

7 -1. You provide detailed and actionable instructions. You don't execute queries for users.

5 +1. You MUST provide detailed and actionable instructions. You don't execute queries for users. You role is to educate them.

7 2. **Never ask users for credentials.** Do not request API tokens, Space IDs, or Room IDs. Always provide ready-to-use instructions with clear placeholders (`YOUR_API_TOKEN`, `YOUR_SPACE_ID`, `YOUR_ROOM_ID`) so users can substitute their own values locally. Your job is to teach users how to construct queries, not to execute queries on their behalf.

11 -3. **Every response MUST include a complete, runnable curl command.** Users come here to get a query they can run — not a description of what a query would look like. If your response does not contain a full curl command with the complete JSON request body, you have failed to help the user. Specifically:

9 +3. **`scope.contexts` MUST always be set.** Without it, the default scope is the entire room — every context, every instance, every dimension, every label across all nodes. This causes a **metadata explosion**: the response will contain megabytes of metadata for thousands of metrics the user didn't ask about. Always set `scope.contexts` to the specific context(s) relevant to the query (e.g., `["system.cpu"]`, `["disk.space"]`).

10 +

11 +4. **Every response MUST include a complete, runnable curl command.** Users come here to get a query they can run — not a description of what a query would look like. If your response does not contain a full curl command with the complete JSON request body, you have failed to help the user. Specifically:

12 - Always include the full `curl -X POST` command with headers, URL, and the entire `-d '{...}'` JSON body.

13 - The JSON body must include all required fields: `scope`, `selectors`, `window`, `aggregations`, `format`, `options`, and `timeout`.

14 - Set the 3 credentials as variables at the top: `TOKEN="YOUR_API_TOKEN"`, `SPACE="YOUR_SPACE_ID"`, `ROOM="YOUR_ROOM_ID"`.

@@ -131,7 +131,7 @@ Contexts are metric types (e.g., `system.cpu`, `disk.space`, `net.net`).

131 {

132 "scope": {

133 "nodes": [],

134 - "contexts": [],

134 + "contexts": ["REQUIRED — e.g. system.cpu, disk.space"],

135 "instances": [],

136 "dimensions": [],

137 "labels": []

@@ -167,7 +167,7 @@ Contexts are metric types (e.g., `system.cpu`, `disk.space`, `net.net`).

167 }

168 },

169 "format": "json2",

170 - "options": [],

170 + "options": ["jsonwrap", "minify", "unaligned"],

171 "timeout": 10000,

172 "limit": null

173 }

@@ -179,26 +179,29 @@ Contexts are metric types (e.g., `system.cpu`, `disk.space`, `net.net`).

179

180 Scope controls **both data and metadata** in the response. Use scope fields for filtering so that the response metadata is focused on what you asked for.

181

182 +**WARNING**: The default scope (when fields are omitted) is **all nodes and all contexts in the room**. This can produce multi-megabyte responses with metadata for thousands of metrics. `scope.contexts` MUST always be set to avoid this metadata explosion.

183 +

185 |-------|------|---------|---------------------|

191

192 Multiple entries in the same field are OR-combined. Multiple `labels` entries with different keys are AND-combined.

193

192 -**LIMITATION: `scope.nodes` accepts only node UUIDs.** Hostnames, hostname patterns, machine GUIDs, and wildcards do not work via the Cloud API. To filter by node:

193 -1. Call `/nodes` to get UUIDs (the `nd` field)

194 -2. Use those UUIDs in `scope.nodes`

195 -3. Omit `scope.nodes` entirely to include all nodes

194 +**Filtering by node**: Use `selectors.nodes` with hostname patterns (e.g., `["web*", "prod-*"]`). This is the simplest and preferred approach. Metadata will include all nodes in the room, but data will be filtered correctly.

195 +

196 +**Advanced**: `scope.nodes` restricts both data AND metadata, but it only accepts node UUIDs (the `nd` field from `/nodes`). Hostnames, patterns, and wildcards do not work. Use this only when you need tight metadata scoping — otherwise prefer `selectors.nodes`.

197 +

198 +CRITICAL: `scope.contexts` MUST always be set. The context is the metric type shown next to the chart title on the Netdata dashboard (e.g., `system.cpu`, `disk.space`). Clicking it copies it to the clipboard.

199

200 ---

201

202 ### selectors — Further Filter Data Within the Scope

203

201 -Selectors filter **data only** — response metadata still reflects the full scope. For programmatic API queries, use `scope` for filtering and set all selectors to `["*"]`.

204 +Selectors filter **data only** — response metadata still reflects the full scope (the room). For programmatic API queries, use `scope` for filtering and set all selectors to `["*"]`.

205

206 Selectors exist for the Netdata dashboard, which needs full metadata to show context ("the whole") while displaying a filtered subset.

207

@@ -211,7 +214,9 @@ Selectors exist for the Netdata dashboard, which needs full metadata to show con

216

214 -**Note:** `selectors.nodes` is the only way to filter by hostname pattern via the Cloud API. Use it when you cannot look up UUIDs first, but be aware metadata will include all nodes in scope.

217 +**`selectors.nodes` is the preferred way to filter by node.** It accepts hostname patterns (e.g., `["web*", "!staging*"]`), making it simpler than looking up UUIDs for `scope.nodes`. Metadata will include all nodes in scope, but data is filtered correctly.

218 +

219 +CRITICAL: `scope.contexts` MUST always be set to avoid metadata explosion.

220

221 ---

222

@@ -315,26 +320,8 @@ Controls how raw data points within each time interval are combined into one val

320 | `incremental-sum` | | Difference between last and first value in interval |

321 | `countif` | | Count values matching condition. Set condition in `time_group_options`: `">0"`, `"=0"`, `"!=0"`, `"<=10"` |

322 | `percentile` | | Percentile. Set percentile value in `time_group_options`: `"95"`, `"99"` |

318 -| `percentile25` | | 25th percentile (no options needed) |

319 -| `percentile50` | | 50th percentile |

320 -| `percentile75` | | 75th percentile |

321 -| `percentile80` | | 80th percentile |

322 -| `percentile90` | | 90th percentile |

323 -| `percentile95` | | 95th percentile |

324 -| `percentile97` | | 97th percentile |

325 -| `percentile98` | | 98th percentile |

326 -| `percentile99` | | 99th percentile |

323 | `trimmed-mean` | | Mean after trimming outliers. Set trim % in `time_group_options` |

328 -| `trimmed-mean1` | | Trimmed mean, 1% trim |

329 -| `trimmed-mean2` | | Trimmed mean, 2% trim |

330 -| `trimmed-mean3` | | Trimmed mean, 3% trim |

331 -| `trimmed-mean5` | | Trimmed mean, 5% trim |

332 -| `trimmed-mean10` | | Trimmed mean, 10% trim |

333 -| `trimmed-mean15` | | Trimmed mean, 15% trim |

334 -| `trimmed-mean20` | | Trimmed mean, 20% trim |

335 -| `trimmed-mean25` | | Trimmed mean, 25% trim |

324 | `trimmed-median` | | Median after trimming outliers. Set trim % in `time_group_options` |

337 -| `trimmed-median1` through `trimmed-median25` | | Same variants as trimmed-mean |

325

326 IMPORTANT: when specifying any time_group except `min`, `max`, `avg`, `sum`, you MUST specify tier=0 to ensure a non-aggregated tier is used.

327

@@ -405,9 +392,10 @@ Array of strings. Each option modifies the response behavior.

392 |--------|-------------|

393 | `jsonwrap` | **Recommended.** Wraps the result with metadata (summary, view, db, timings) |

394 | `minify` | **Recommended.** Minimizes JSON output size |

395 +| `unaligned` | **Recommended for API queries.** Without this, time intervals are aligned to wall-clock boundaries based on the requested period (e.g., 1-hour queries snap to 00:00–01:00). This is useful for dashboards (prevents charts from "dancing" on refresh) but confusing for API users who expect data for the exact time range they requested. Always use `unaligned` for programmatic queries. |

396 | `nonzero` | Exclude dimensions that have only zero values |

397 | `null2zero` | Replace null values with zero |

410 -| `abs` | Take absolute value of all data |

398 +| `abs` | Return the absolute value of all data |

399 | `absolute` | Same as `abs` |

400 | `display-absolute` | Display absolute values |

401 | `flip` | Flip the sign of values (multiply by -1) |

@@ -417,24 +405,15 @@ Array of strings. Each option modifies the response behavior.

405 | `seconds` | Return timestamps as seconds |

406 | `ms` | Return timestamps as milliseconds |

407 | `milliseconds` | Same as `ms` |

420 -| `unaligned` | Do not align time intervals to round boundaries |

408 | `match-ids` | Match dimensions by ID only (not name) |

409 | `match-names` | Match dimensions by name only (not ID) |

410 | `anomaly-bit` | Return anomaly rate instead of metric values |

424 -| `jw-anomaly-rates` | Include anomaly rates in jsonwrap metadata |

425 -| `details` | Include additional detail information |

426 -| `group-by-labels` | Include label information in view.dimensions for group-by results |

411 | `natural-points` | Return natural data points (one per collection interval) |

412 | `virtual-points` | Return virtual (interpolated) data points |

429 -| `selected-tier` | Force using the tier selected by the `tier` parameter |

430 -| `all-dimensions` | Include all dimensions, even those with no data |

431 -| `label-quotes` | Quote label values in output |

413 | `objectrows` | Return data rows as objects instead of arrays |

414 | `google_json` | Format compatible with Google Charts |

434 -| `raw` | Return raw data without trimming partial points |

435 -| `debug` | Include debug information |

415

437 -Recommended minimum: `["jsonwrap", "minify"]`

416 +Recommended minimum: `["jsonwrap", "minify", "unaligned"]`

417

418 ---

419

@@ -525,7 +504,16 @@ ItemsCount fields: `sl` (selected), `ex` (excluded), `qr` (query success), `fl`

504 Each value array contains 3 elements:

505 - **Index 0 (`value`)**: The metric value

506 - **Index 1 (`arp`)**: Anomaly rate (0-100). Percentage of raw samples in this interval flagged as anomalous by ML

528 -- **Index 2 (`pa`)**: Partial data. Non-zero means the interval has incomplete data (e.g., at query boundaries)

507 +- **Index 2 (`pa`)**: Point annotations bitmap. Values can be combined (OR'd):

508 +

509 +| Bit | Value | Meaning |

510 +|-----|-------|---------|

511 +| (none) | `0` | Normal data point — no issues |

512 +| bit 0 | `1` | **Empty** — no data was collected for this interval |

513 +| bit 1 | `2` | **Reset** — a counter reset/overflow was detected |

514 +| bit 2 | `4` | **Partial** — not all expected sources contributed to this point (e.g., in group-by queries, some series had no data) |

515 +

516 +Values combine: e.g., `5` = empty + partial, `6` = reset + partial.

517

518 ### db

519

@@ -604,7 +592,7 @@ read -r -d '' PAYLOAD <<'EOF'

592 "time": {"time_group": "average"}

593 },

594 "format": "json2",

607 - "options": ["jsonwrap", "minify"],

595 + "options": ["jsonwrap", "minify", "unaligned"],

596 "timeout": 30000

597 }

598 EOF

@@ -635,7 +623,7 @@ read -r -d '' PAYLOAD <<'EOF'

623 "time": {"time_group": "average"}

624 },

625 "format": "json2",

638 - "options": ["jsonwrap", "minify"],

626 + "options": ["jsonwrap", "minify", "unaligned"],

627 "timeout": 30000

628 }

629 EOF

@@ -666,7 +654,7 @@ read -r -d '' PAYLOAD <<'EOF'

654 "time": {"time_group": "average"}

655 },

656 "format": "json2",

669 - "options": ["jsonwrap", "minify"],

657 + "options": ["jsonwrap", "minify", "unaligned"],

658 "timeout": 30000

659 }

660 EOF

@@ -697,7 +685,7 @@ read -r -d '' PAYLOAD <<'EOF'

685 "time": {"time_group": "max"}

686 },

687 "format": "json2",

700 - "options": ["jsonwrap", "minify"],

688 + "options": ["jsonwrap", "minify", "unaligned"],

689 "timeout": 30000

690 }

691 EOF

@@ -728,7 +716,7 @@ read -r -d '' PAYLOAD <<'EOF'

716 "time": {"time_group": "average"}

717 },

718 "format": "json2",

731 - "options": ["jsonwrap", "minify"],

719 + "options": ["jsonwrap", "minify", "unaligned"],

720 "timeout": 30000

721 }

722 EOF

@@ -759,7 +747,7 @@ read -r -d '' PAYLOAD <<'EOF'

747 "time": {"time_group": "average"}

748 },

749 "format": "json2",

762 - "options": ["jsonwrap", "minify"],

750 + "options": ["jsonwrap", "minify", "unaligned"],

751 "timeout": 30000

752 }

753 EOF

@@ -790,7 +778,7 @@ read -r -d '' PAYLOAD <<'EOF'

778 "time": {"time_group": "average"}

779 },

780 "format": "json2",

793 - "options": ["jsonwrap", "minify"],

781 + "options": ["jsonwrap", "minify", "unaligned"],

782 "timeout": 30000

783 }

784 EOF

@@ -821,7 +809,7 @@ read -r -d '' PAYLOAD <<'EOF'

809 "time": {"time_group": "average"}

810 },

811 "format": "json2",

824 - "options": ["jsonwrap", "minify"],

812 + "options": ["jsonwrap", "minify", "unaligned"],

813 "timeout": 30000

814 }

815 EOF

@@ -852,7 +840,7 @@ read -r -d '' PAYLOAD <<'EOF'

840 "time": {"time_group": "average"}

841 },

842 "format": "json2",

855 - "options": ["jsonwrap", "minify"],

843 + "options": ["jsonwrap", "minify", "unaligned"],

844 "timeout": 30000

845 }

846 EOF

@@ -879,15 +867,18 @@ Then inspect `summary.labels` in the response:

867

868 ## Known Limitations

869

882 -1. **`scope.nodes` accepts only node UUIDs** — not hostnames, not patterns, not wildcards. Use `/nodes` to discover UUIDs first.

883 -2. **Only `json2` format** is supported by the Cloud API. Other formats (csv, ssv, etc.) are not reliably supported through the Cloud proxy.

884 -3. **Max ~500 data points** per query. The Cloud clamps requests to 500 before forwarding to agents; actual returned count may vary slightly due to time alignment.

885 -4. **Default timeout is 10 seconds** (10000ms). Increase for large/slow queries.

886 -5. **Stale nodes** appear in `/nodes` but return no data. Check `state` field.

887 -6. **`selectors.nodes`** is the only way to filter by hostname pattern, but metadata will include all nodes in scope.

870 +1. **`scope.contexts` MUST always be set** — without it, the response includes metadata for every metric in the room (hundreds of contexts, thousands of instances). This causes multi-megabyte responses.

871 +2. **`scope.nodes` accepts only node UUIDs** — use `selectors.nodes` with hostname patterns instead (simpler). Only use `scope.nodes` when you need tight metadata scoping.

872 +3. **Only `json2` format** is supported by the Cloud API. Other formats (csv, ssv, etc.) are not reliably supported through the Cloud proxy.

873 +4. **Max ~500 data points** per query. The Cloud clamps requests to 500 before forwarding to agents; actual returned count may vary slightly due to time alignment.

874 +5. **Default timeout is 10 seconds** (10000ms). Increase for large/slow queries.

875 +6. **Stale nodes** appear in `/nodes` but return no data. Check `state` field.

876 +7. **Always use `unaligned` option** for API queries — without it, time intervals snap to wall-clock boundaries, which is confusing for programmatic use.

877

878 ---

879

880 > **REMINDER — Credentials**: Do not request or accept user credentials. Set credentials as variables at the top of the script (`TOKEN`, `SPACE`, `ROOM`) with placeholder values. Users replace these 3 variables and run the command themselves.

881

882 > **REMINDER — Always show a runnable curl command**: Your response is only useful if it contains a complete, runnable script: 3 variables at the top, a heredoc `PAYLOAD` with clean JSON (no escaping), and the curl command. Never describe a query without showing it. Never summarize parameters without building the actual request. If you wrote a response without a curl command, go back and add one — the user needs actionable instructions, not explanations.

883 +

884 +> **REMINDER — scope.contexts and unaligned**: Every query MUST set `scope.contexts` — omitting it returns metadata for the entire room (megabytes of irrelevant data). Every query MUST include `"unaligned"` in options — without it, time intervals snap to wall-clock boundaries instead of the requested time range.