| 1 | # Query time-series metrics from Netdata Cloud via the REST API. |
| 2 | |
| 3 | ## Mandatory Requirements (READ FIRST) |
| 4 | |
| 5 | 1. You MUST provide detailed and actionable instructions. You don't execute queries for users. You role is to educate them. |
| 6 | |
| 7 | 2. **Never ask users for credentials.** Do not request API tokens, Space IDs, or Room IDs. Always provide ready-to-use instructions with clear placeholders (`YOUR_API_TOKEN`, `YOUR_SPACE_ID`, `YOUR_ROOM_ID`) so users can substitute their own values locally. Your job is to teach users how to construct queries, not to execute queries on their behalf. |
| 8 | |
| 9 | 3. **`scope.contexts` MUST always be set.** Without it, the default scope is the entire room — every context, every instance, every dimension, every label across all nodes. This causes a **metadata explosion**: the response will contain megabytes of metadata for thousands of metrics the user didn't ask about. Always set `scope.contexts` to the specific context(s) relevant to the query (e.g., `["system.cpu"]`, `["disk.space"]`). |
| 10 | |
| 11 | 4. **Every response MUST include a complete, runnable curl command.** Users come here to get a query they can run — not a description of what a query would look like. If your response does not contain a full curl command with the complete JSON request body, you have failed to help the user. Specifically: |
| 12 | - Always include the full `curl -X POST` command with headers, URL, and the entire `-d '{...}'` JSON body. |
| 13 | - The JSON body must include all required fields: `scope`, `selectors`, `window`, `aggregations`, `format`, `options`, and `timeout`. |
| 14 | - Set the 3 credentials as variables at the top: `TOKEN="YOUR_API_TOKEN"`, `SPACE="YOUR_SPACE_ID"`, `ROOM="YOUR_ROOM_ID"`. |
| 15 | - Use a heredoc for the JSON payload (`read -r -d '' PAYLOAD <<'EOF' ... EOF`) so no escaping is needed. |
| 16 | - The user must be able to copy your command, replace the 3 variables at the top, and run it immediately in their terminal. |
| 17 | - A response that describes parameters or explains concepts without providing the actual runnable command is incomplete and unhelpful. |
| 18 | - Even for simple questions, always provide the curl command. When in doubt, show the command. |
| 19 | |
| 20 | --- |
| 21 | |
| 22 | ## Prerequisites |
| 23 | |
| 24 | Three things are needed: |
| 25 | |
| 26 | ### 1. API Token |
| 27 | |
| 28 | 1. Login to [app.netdata.cloud](https://app.netdata.cloud) |
| 29 | 2. Click user icon (lower-left corner, tooltip shows your name) |
| 30 | 3. Select **User Settings** |
| 31 | 4. In the modal, select the **API Tokens** tab |
| 32 | 5. Click the **[+]** button (top-left) |
| 33 | 6. Select a scope, enter a description, click **Create** |
| 34 | 7. **Copy the token immediately** — it will not be shown again |
| 35 | |
| 36 | Relevant scopes: `scope:all` (full access), `scope:grafana-plugin` (data endpoints). |
| 37 | |
| 38 | ### 2. Space ID |
| 39 | |
| 40 | 1. In the dashboard left side, at the spaces list, click the **gear icon** below the spaces list (tooltip: "Space Settings") |
| 41 | 2. In the **Info** tab, copy the **Space Id** |
| 42 | |
| 43 | ### 3. Room ID |
| 44 | |
| 45 | 1. In the same Space Settings, go to the **Rooms** tab |
| 46 | 2. Find the room, click the **>** icon at the right of the room row (tooltip: "Room Settings") |
| 47 | 3. In the **Room** tab, copy the **Room Id** |
| 48 | |
| 49 | --- |
| 50 | |
| 51 | ## API Endpoints |
| 52 | |
| 53 | Base URL: `https://app.netdata.cloud` |
| 54 | Swagger online: https://app.netdata.cloud/api/docs/ |
| 55 | |
| 56 | All endpoints use **POST** with a JSON body and require: |
| 57 | |
| 58 | ``` |
| 59 | Authorization: Bearer YOUR_API_TOKEN |
| 60 | Content-Type: application/json |
| 61 | ``` |
| 62 | |
| 63 | | Endpoint | Purpose | |
| 64 | |----------|---------| |
| 65 | | `/api/v3/spaces/{spaceID}/rooms/{roomID}/data` | Query time-series data | |
| 66 | | `/api/v3/spaces/{spaceID}/rooms/{roomID}/nodes` | List nodes in the room | |
| 67 | | `/api/v3/spaces/{spaceID}/rooms/{roomID}/contexts` | List available metric contexts | |
| 68 | |
| 69 | --- |
| 70 | |
| 71 | ## Discover Nodes |
| 72 | |
| 73 | **Endpoint:** POST `/api/v3/spaces/{spaceID}/rooms/{roomID}/nodes` |
| 74 | **Body:** `{}` |
| 75 | |
| 76 | Response fields per node: |
| 77 | |
| 78 | | JSON field | Description | |
| 79 | |------------|-------------| |
| 80 | | `nd` | **Node UUID** — required for `scope.nodes` in data queries | |
| 81 | | `nm` | Hostname | |
| 82 | | `mg` | Machine GUID | |
| 83 | | `state` | `reachable` (live) or `stale` (disconnected) | |
| 84 | | `v` | Agent version | |
| 85 | | `labels` | All node labels as key-value pairs | |
| 86 | | `hw` | Hardware: `cpus`, `memory`, `disk_space`, `architecture` | |
| 87 | | `os` | OS: `nm` (name), `v` (version), `kernel` | |
| 88 | | `health` | Alert summary: `status`, `alerts.warning`, `alerts.critical` | |
| 89 | | `capabilities` | Supported features: `ml`, `funcs`, `health`, etc. | |
| 90 | |
| 91 | Example: |
| 92 | |
| 93 | ```bash |
| 94 | TOKEN="YOUR_API_TOKEN" |
| 95 | SPACE="YOUR_SPACE_ID" |
| 96 | ROOM="YOUR_ROOM_ID" |
| 97 | |
| 98 | curl -s -X POST \ |
| 99 | -H 'Content-Type: application/json' \ |
| 100 | -H "Authorization: Bearer $TOKEN" \ |
| 101 | "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/nodes" \ |
| 102 | -d '{}' |
| 103 | ``` |
| 104 | |
| 105 | --- |
| 106 | |
| 107 | ## Discover Contexts |
| 108 | |
| 109 | Contexts are metric types (e.g., `system.cpu`, `disk.space`, `net.net`). |
| 110 | |
| 111 | **Endpoint:** POST `/api/v3/spaces/{spaceID}/rooms/{roomID}/contexts` |
| 112 | |
| 113 | ```json |
| 114 | { |
| 115 | "scope": { "contexts": ["system.*"] }, |
| 116 | "selectors": { "nodes": ["*"], "contexts": ["*"] } |
| 117 | } |
| 118 | ``` |
| 119 | |
| 120 | `scope.contexts` supports patterns: `system.*`, `disk.*`, `*cpu*`. |
| 121 | |
| 122 | --- |
| 123 | |
| 124 | ## Query Metric Data |
| 125 | |
| 126 | **Endpoint:** POST `/api/v3/spaces/{spaceID}/rooms/{roomID}/data` |
| 127 | |
| 128 | ### Full Request Body Structure |
| 129 | |
| 130 | ```json |
| 131 | { |
| 132 | "scope": { |
| 133 | "nodes": [], |
| 134 | "contexts": ["REQUIRED — e.g. system.cpu, disk.space"], |
| 135 | "instances": [], |
| 136 | "dimensions": [], |
| 137 | "labels": [] |
| 138 | }, |
| 139 | "selectors": { |
| 140 | "nodes": ["*"], |
| 141 | "contexts": ["*"], |
| 142 | "instances": ["*"], |
| 143 | "dimensions": ["*"], |
| 144 | "labels": ["*"], |
| 145 | "alerts": ["*"] |
| 146 | }, |
| 147 | "window": { |
| 148 | "after": 0, |
| 149 | "before": 0, |
| 150 | "points": 0, |
| 151 | "duration": 0, |
| 152 | "tier": null, |
| 153 | "baseline": null |
| 154 | }, |
| 155 | "aggregations": { |
| 156 | "metrics": [ |
| 157 | { |
| 158 | "group_by": [], |
| 159 | "group_by_label": [], |
| 160 | "aggregation": "avg" |
| 161 | } |
| 162 | ], |
| 163 | "time": { |
| 164 | "time_group": "average", |
| 165 | "time_group_options": null, |
| 166 | "time_resampling": null |
| 167 | } |
| 168 | }, |
| 169 | "format": "json2", |
| 170 | "options": ["jsonwrap", "minify", "unaligned"], |
| 171 | "timeout": 10000, |
| 172 | "limit": null |
| 173 | } |
| 174 | ``` |
| 175 | |
| 176 | --- |
| 177 | |
| 178 | ### scope — Define the Data Universe |
| 179 | |
| 180 | Scope controls **both data and metadata** in the response. Use scope fields for filtering so that the response metadata is focused on what you asked for. |
| 181 | |
| 182 | **WARNING**: The default scope (when fields are omitted) is **all nodes and all contexts in the room**. This can produce multi-megabyte responses with metadata for thousands of metrics. `scope.contexts` MUST always be set to avoid this metadata explosion. |
| 183 | |
| 184 | | Field | Type | Accepts | Default (if omitted) | |
| 185 | |-------|------|---------|---------------------| |
| 186 | | `nodes` | `string[]` | **Node UUIDs only** (the `nd` field from `/nodes`) | All nodes in the room | |
| 187 | | `contexts` | `string[]` | Exact names or patterns (`system.*`, `*cpu*`) | **REQUIRED** — always set to avoid metadata explosion | |
| 188 | | `instances` | `string[]` | Exact names or patterns (`disk_space./@NODE_UUID`) | All instances | |
| 189 | | `dimensions` | `string[]` | Exact names or patterns (`*user*`, `sent`) | All dimensions | |
| 190 | | `labels` | `string[]` | `key:value` pairs (`filesystem:btrfs`, `mount_point:/`) | No label filter | |
| 191 | |
| 192 | Multiple entries in the same field are OR-combined. Multiple `labels` entries with different keys are AND-combined. |
| 193 | |
| 194 | **Filtering by node**: Use `selectors.nodes` with hostname patterns (e.g., `["web*", "prod-*"]`). This is the simplest and preferred approach. Metadata will include all nodes in the room, but data will be filtered correctly. |
| 195 | |
| 196 | **Advanced**: `scope.nodes` restricts both data AND metadata, but it only accepts node UUIDs (the `nd` field from `/nodes`). Hostnames, patterns, and wildcards do not work. Use this only when you need tight metadata scoping — otherwise prefer `selectors.nodes`. |
| 197 | |
| 198 | CRITICAL: `scope.contexts` MUST always be set. The context is the metric type shown next to the chart title on the Netdata dashboard (e.g., `system.cpu`, `disk.space`). Clicking it copies it to the clipboard. |
| 199 | |
| 200 | --- |
| 201 | |
| 202 | ### selectors — Further Filter Data Within the Scope |
| 203 | |
| 204 | Selectors filter **data only** — response metadata still reflects the full scope (the room). For programmatic API queries, use `scope` for filtering and set all selectors to `["*"]`. |
| 205 | |
| 206 | Selectors exist for the Netdata dashboard, which needs full metadata to show context ("the whole") while displaying a filtered subset. |
| 207 | |
| 208 | | Field | Type | Checked against | Supports | |
| 209 | |-------|------|----------------|----------| |
| 210 | | `nodes` | `string[]` | Machine GUID, node ID, **hostname** | Simple patterns, positive and negative | |
| 211 | | `contexts` | `string[]` | Context ID | Simple patterns, positive and negative | |
| 212 | | `instances` | `string[]` | Instance ID, instance name, `instance@machine_guid` | Simple patterns, positive and negative | |
| 213 | | `dimensions` | `string[]` | Dimension ID and dimension name | Simple patterns, positive and negative | |
| 214 | | `labels` | `string[]` | `name:value` of all labels | Simple patterns (negative not recommended) | |
| 215 | | `alerts` | `string[]` | Alert name, `name:status` (CLEAR, WARNING, CRITICAL, REMOVED, UNDEFINED, UNINITIALIZED) | Simple patterns; negative excludes instances | |
| 216 | |
| 217 | **`selectors.nodes` is the preferred way to filter by node.** It accepts hostname patterns (e.g., `["web*", "!staging*"]`), making it simpler than looking up UUIDs for `scope.nodes`. Metadata will include all nodes in scope, but data is filtered correctly. |
| 218 | |
| 219 | CRITICAL: `scope.contexts` MUST always be set to avoid metadata explosion. |
| 220 | |
| 221 | --- |
| 222 | |
| 223 | ### window — Time Range |
| 224 | |
| 225 | | Field | Type | Description | Default | |
| 226 | |-------|------|-------------|---------| |
| 227 | | `after` | `int` | Start time. Negative = relative seconds from `before` (max -94608000 = 3 years). Positive = Unix epoch. | `-600` | |
| 228 | | `before` | `int` | End time. Negative = relative seconds from now (max -94608000). Positive = Unix epoch. | `0` (now) | |
| 229 | | `points` | `int` | Number of data points to return. `0` or omitted = all available points. | `0` | |
| 230 | | `duration` | `int` | Alternative to after/before. Duration in seconds. | `0` | |
| 231 | | `tier` | `int?` | Force a specific dbengine storage tier (0 = per-second, 1 = per-minute, 2 = per-hour). `null` = auto-select. | `null` | |
| 232 | | `baseline` | `object?` | Baseline window for comparison queries. Same fields as window: `after`, `before`, `points`, `duration`. | `null` | |
| 233 | |
| 234 | Max points requested: approximately **500** (`ScopeDataRequestMaxPoints`). The Cloud clamps the request to 500 before forwarding to agents, but the actual number returned may vary slightly due to time alignment. |
| 235 | |
| 236 | The time range is divided into `points` equal intervals. Each interval is aggregated using the `time_group` function. |
| 237 | |
| 238 | --- |
| 239 | |
| 240 | ### Time resolution: `duration ÷ points = seconds per point` |
| 241 | |
| 242 | This is the most common assistant mistake. The number of `points` does NOT mean "give me per-second data". It means "split the duration into N equal buckets". The actual time resolution per point is: |
| 243 | |
| 244 | ``` |
| 245 | seconds_per_point = abs(duration) ÷ points |
| 246 | ``` |
| 247 | |
| 248 | **To get per-second data, set `points` equal to the duration in seconds.** |
| 249 | |
| 250 | Examples: |
| 251 | |
| 252 | | You want | Set `after` | Set `points` | Result | |
| 253 | |---|---|---|---| |
| 254 | | Per-second resolution, last 2 minutes | `-120` | `120` | 1 second per point | |
| 255 | | Per-second resolution, last 5 minutes | `-300` | `300` | 1 second per point | |
| 256 | | 10-second buckets, last 10 minutes | `-600` | `60` | 10 seconds per point | |
| 257 | | Per-minute resolution, last hour | `-3600` | `60` | 60 seconds per point | |
| 258 | |
| 259 | **Common mistake**: requesting `after: -600, points: 30` and expecting per-second data. Result: 600 ÷ 30 = **20 seconds per point** (heavily aggregated). Per-second data over 10 minutes requires `after: -600, points: 600` (which is at the 500-point cap; either request 8 minutes 20 seconds at 500 points, or accept a slightly coarser resolution). |
| 260 | |
| 261 | **Per-second data also requires that the dbengine tier 0 (per-second storage) covers the requested time range.** If the agent's tier 0 retention is shorter than `abs(after)`, the engine auto-selects a coarser tier (per-minute or per-hour). Force tier 0 with `"tier": 0` in the window if you need to assert per-second data is actually available -- the query will fail rather than silently downsample. |
| 262 | |
| 263 | **`points: 0` (the default) is NOT "per-second"** -- it requests "all available points", which is whatever the engine returns within its 500-point cap and the storage tier's natural granularity. For a 1-hour query against tier-0 storage, the engine still aggregates because 3600 > 500. |
| 264 | |
| 265 | --- |
| 266 | |
| 267 | ### How the Query Pipeline Works |
| 268 | |
| 269 | The query engine is a pipeline with two aggregation stages: |
| 270 | |
| 271 | 1. **Identify time-series** matching the `scope` and `selectors` |
| 272 | 2. **Set up the output time-series** based on `group_by` (e.g., 2 groups for label values A and B) |
| 273 | 3. **For each matched time-series:** |
| 274 | - **Stage 1 — Time aggregation** (`time_group`): Aggregate raw samples within each time interval into `points` data points (e.g., average 86400 per-second samples into 100 points) |
| 275 | - **Stage 2 — Metric aggregation** (`aggregation`): Add the time-aggregated points into the appropriate output time-series using the aggregation function (e.g., SUM into group A or B) |
| 276 | 4. **Present** the grouped, aggregated result |
| 277 | |
| 278 | **Key insight**: `time_group` reduces samples within each time-series. `aggregation` combines multiple time-series into groups. They operate in sequence — metric aggregation works on already time-aggregated data. |
| 279 | |
| 280 | #### Example: 1000 containers, group by label `namespace` (2 values: A and B), 100 points over 1 day |
| 281 | |
| 282 | 1. Output setup: 2 time-series needed (A and B), each with 100 points |
| 283 | 2. For each of the 1000 container time-series: |
| 284 | - Time-aggregate 86400 seconds into 100 points using `time_group` (e.g., `average`) |
| 285 | - Add those 100 points into either A or B using `aggregation` (e.g., `sum`) |
| 286 | 3. Result: 2 columns (A, B) × 100 rows |
| 287 | |
| 288 | #### Choosing time_group Based on What the User Wants |
| 289 | |
| 290 | | User intent | time_group | Why | |
| 291 | |-------------|-----------|-----| |
| 292 | | Average resource consumption (rate metrics: CPU, I/O, bandwidth) | `average` | Rate metrics represent per-second rates; averaging preserves the rate | |
| 293 | | Average resource consumption (gauge metrics: memory, disk space, connections) | `average` or `max` | Gauges represent current state; max shows peak usage | |
| 294 | | Find spikes or peaks (any metric type) | `max` | Captures the highest value within each interval | |
| 295 | | Total volume transferred (counters: bytes, packets) | `sum` | Sums the actual volume | |
| 296 | | Count events matching a condition | `countif` | Counts samples matching a threshold | |
| 297 | |
| 298 | #### Choosing aggregation Based on How to Combine Series |
| 299 | |
| 300 | | User intent | aggregation | Why | |
| 301 | |-------------|------------|-----| |
| 302 | | Total across all series (e.g., total CPU across all containers) | `sum` | Adds up all contributions | |
| 303 | | Average across series | `avg` | Mean of the group | |
| 304 | | Worst case across series | `max` | Highest value in the group | |
| 305 | | Best case across series | `min` | Lowest value in the group | |
| 306 | |
| 307 | #### Mapping User Questions to Parameters |
| 308 | |
| 309 | **"Find a CPU spike over the last week across all my containers"** |
| 310 | → `time_group: "max"`, `aggregation: "sum"` (sum user+system), `group_by: ["instance"]` |
| 311 | |
| 312 | **"Which namespace consumed most CPU over the last week?"** |
| 313 | → `time_group: "average"` (per-second rate) or `"sum"` (total), `aggregation: "sum"`, `group_by: ["label"]`, `group_by_label: ["namespace"]` |
| 314 | |
| 315 | **"Peak memory usage per node over the last 24 hours"** |
| 316 | → `time_group: "max"` (gauge metric, want peak), `aggregation: "sum"`, `group_by: ["node"]` |
| 317 | |
| 318 | #### Research the Context Before Answering |
| 319 | |
| 320 | Before constructing a query for a user, you should understand the metric context they are asking about — its dimensions, labels, and whether it represents rates (`incremental`) or gauges (`absolute`). Search for the context name (e.g., `cgroup.cpu`, `disk.space`, `nginx.connections`) in the Netdata source code to find its `metadata.yaml`, which defines dimensions, units, chart type, and available labels. This ensures you choose the correct `time_group` and `aggregation` for their use case. |
| 321 | |
| 322 | --- |
| 323 | |
| 324 | ### aggregations.time — Time Aggregation |
| 325 | |
| 326 | Controls how raw data points within each time interval are combined into one value per series. |
| 327 | |
| 328 | | Field | Type | Description | Default | |
| 329 | |-------|------|-------------|---------| |
| 330 | | `time_group` | `string` | Aggregation function (see table below) | `average` | |
| 331 | | `time_group_options` | `string?` | Additional parameter for the function | `null` | |
| 332 | | `time_resampling` | `int?` | Resample "per-second" values to "per-minute" (60) or "per-hour" (3600). Only works with `time_group=average`. | `null` | |
| 333 | |
| 334 | #### time_group values |
| 335 | |
| 336 | | Value | Aliases | Description | |
| 337 | |-------|---------|-------------| |
| 338 | | `average` | `avg` | Mean value **(default)** | |
| 339 | | `min` | | Minimum value | |
| 340 | | `max` | | Maximum value | |
| 341 | | `sum` | | Sum of values | |
| 342 | | `median` | | Median value | |
| 343 | | `stddev` | | Standard deviation | |
| 344 | | `cv` | | Coefficient of variation (stddev/mean) | |
| 345 | | `ses` | | Single exponential smoothing | |
| 346 | | `des` | | Double exponential smoothing | |
| 347 | | `incremental-sum` | | Difference between last and first value in interval | |
| 348 | | `countif` | | Count values matching condition. Set condition in `time_group_options`: `">0"`, `"=0"`, `"!=0"`, `"<=10"` | |
| 349 | | `percentile` | | Percentile. Set percentile value in `time_group_options`: `"95"`, `"99"` | |
| 350 | | `trimmed-mean` | | Mean after trimming outliers. Set trim % in `time_group_options` | |
| 351 | | `trimmed-median` | | Median after trimming outliers. Set trim % in `time_group_options` | |
| 352 | |
| 353 | :::important |
| 354 | |
| 355 | When using `time_group` values other than `min`, `max`, `average`, or `sum`, you MUST specify `"tier": 0` in the `window` object to ensure a non-aggregated storage tier is used. Without it, the query may use a pre-aggregated tier (per-minute or per-hour) where advanced functions like `median`, `stddev`, `ses`, `des`, `percentile`, `countif`, `trimmed-mean`, `trimmed-median`, and `extremes` cannot work correctly. |
| 356 | |
| 357 | ::: |
| 358 | |
| 359 | #### time_group_options values |
| 360 | |
| 361 | | Used with | Value format | Example | |
| 362 | |-----------|-------------|---------| |
| 363 | | `countif` | Comparison operator + value | `">0"`, `"=0"`, `"!=0"`, `"<=100"` | |
| 364 | | `percentile` | Percentile value (0-100) | `"95"`, `"99.5"` | |
| 365 | | `trimmed-mean` | Trim percentage | `"5"`, `"10"` | |
| 366 | | `trimmed-median` | Trim percentage | `"5"`, `"10"` | |
| 367 | |
| 368 | :::important |
| 369 | |
| 370 | When using `time_group` values other than `min`, `max`, `average`, or `sum`, you MUST specify `"tier": 0` in the `window` object to ensure a non-aggregated storage tier is used. Without it, the query may use a pre-aggregated tier (per-minute or per-hour) where advanced functions like `median`, `stddev`, `ses`, `des`, `percentile`, `countif`, `trimmed-mean`, `trimmed-median`, and `extremes` cannot work correctly. |
| 371 | |
| 372 | ::: |
| 373 | |
| 374 | --- |
| 375 | |
| 376 | ### aggregations.metrics[] — Dimension Aggregation |
| 377 | |
| 378 | Controls how multiple time-series are combined. Each entry defines a grouping pass. At least one is required. |
| 379 | |
| 380 | | Field | Type | Description | Default | |
| 381 | |-------|------|-------------|---------| |
| 382 | | `group_by` | `string[]` | What to group by (see table below) | (required) | |
| 383 | | `group_by_label` | `string[]` | Label keys to group by. Required when `group_by` includes `label`. Order is respected. | `[]` | |
| 384 | | `aggregation` | `string` | How to combine grouped values (see table below) | `average` | |
| 385 | |
| 386 | #### group_by values |
| 387 | |
| 388 | All values can be combined together **except** `selected` (if `selected` is present, all others are ignored). |
| 389 | |
| 390 | | Value | Result columns represent | Use case | |
| 391 | |-------|------------------------|----------| |
| 392 | | `selected` | Single column: all matched data combined into one series | Total/aggregate value across everything | |
| 393 | | `dimension` | One column per unique dimension name | Break down by metric component (user/system/iowait for CPU) | |
| 394 | | `node` | One column per node | Compare nodes side by side | |
| 395 | | `instance` | One column per instance (`context@hostname`) | Compare instances across nodes | |
| 396 | | `label` | One column per unique label value | Group by label (requires `group_by_label`) | |
| 397 | | `context` | One column per context | Compare different metric types | |
| 398 | | `units` | One column per unit type | Group by measurement unit | |
| 399 | | `percentage-of-instance` | Percentages per dimension within each instance | Show proportions instead of absolutes | |
| 400 | |
| 401 | Combination example: `"group_by": ["node", "dimension"]` creates one column per node+dimension combination. |
| 402 | |
| 403 | #### aggregation values |
| 404 | |
| 405 | | Value | Aliases | Description | |
| 406 | |-------|---------|-------------| |
| 407 | | `avg` | `average` | Mean of grouped values **(default)** | |
| 408 | | `sum` | | Sum of grouped values | |
| 409 | | `min` | | Minimum of grouped values | |
| 410 | | `max` | | Maximum of grouped values | |
| 411 | | `median` | | Median of grouped values | |
| 412 | | `percentage` | | Express as percentage of total | |
| 413 | |
| 414 | --- |
| 415 | |
| 416 | ### format |
| 417 | |
| 418 | Only `json2` is supported by Netdata Cloud. |
| 419 | |
| 420 | --- |
| 421 | |
| 422 | ### options |
| 423 | |
| 424 | Array of strings. Each option modifies the response behavior. |
| 425 | |
| 426 | | Option | Description | |
| 427 | |--------|-------------| |
| 428 | | `jsonwrap` | **Recommended.** Wraps the result with metadata (summary, view, db, timings) | |
| 429 | | `minify` | **Recommended.** Minimizes JSON output size | |
| 430 | | `unaligned` | **Recommended for API queries.** Without this, time intervals are aligned to wall-clock boundaries based on the requested period (e.g., 1-hour queries snap to 00:00–01:00). This is useful for dashboards (prevents charts from "dancing" on refresh) but confusing for API users who expect data for the exact time range they requested. Always use `unaligned` for programmatic queries. | |
| 431 | | `nonzero` | Exclude dimensions that have only zero values | |
| 432 | | `null2zero` | Replace null values with zero | |
| 433 | | `abs` | Return the absolute value of all data | |
| 434 | | `absolute` | Same as `abs` | |
| 435 | | `display-absolute` | Display absolute values | |
| 436 | | `flip` | Flip the sign of values (multiply by -1) | |
| 437 | | `reversed` | Reverse the order of data points (oldest last) | |
| 438 | | `min2max` | Show the range (max - min) instead of the value | |
| 439 | | `percentage` | Convert values to percentages | |
| 440 | | `seconds` | Return timestamps as seconds | |
| 441 | | `ms` | Return timestamps as milliseconds | |
| 442 | | `milliseconds` | Same as `ms` | |
| 443 | | `match-ids` | Match dimensions by ID only (not name) | |
| 444 | | `match-names` | Match dimensions by name only (not ID) | |
| 445 | | `anomaly-bit` | Return anomaly rate instead of metric values | |
| 446 | | `natural-points` | Return natural data points (one per collection interval) | |
| 447 | | `virtual-points` | Return virtual (interpolated) data points | |
| 448 | | `objectrows` | Return data rows as objects instead of arrays | |
| 449 | | `google_json` | Format compatible with Google Charts | |
| 450 | |
| 451 | Recommended minimum: `["jsonwrap", "minify", "unaligned"]` |
| 452 | |
| 453 | --- |
| 454 | |
| 455 | ### timeout |
| 456 | |
| 457 | Query timeout in milliseconds. Default: `10000` (10 seconds). Set higher for queries spanning many nodes or long time ranges. |
| 458 | |
| 459 | ### limit |
| 460 | |
| 461 | Optional integer. Limits the number of dimensions returned. Cannot be negative. Useful when querying high-cardinality contexts. |
| 462 | |
| 463 | --- |
| 464 | |
| 465 | ## Response Structure |
| 466 | |
| 467 | With `jsonwrap` option, the response contains: |
| 468 | |
| 469 | ### Top-level fields |
| 470 | |
| 471 | | Field | Description | |
| 472 | |-------|-------------| |
| 473 | | `api` | API version (integer) | |
| 474 | | `agents` | List of agents consulted | |
| 475 | | `versions` | Hash values to detect database changes | |
| 476 | | `summary` | Metadata about nodes, contexts, instances, dimensions, labels, alerts | |
| 477 | | `totals` | Counts of selected/excluded/queried items | |
| 478 | | `functions` | List of supported functions | |
| 479 | | `db` | Database info (tiers, retention, update frequency) | |
| 480 | | `view` | Presentation metadata (title, units, dimensions, time range) | |
| 481 | | `result` | **The actual time-series data** | |
| 482 | | `timings` | Query performance metrics | |
| 483 | |
| 484 | ### summary |
| 485 | |
| 486 | Metadata determined by `scope`. Statistics within are influenced by `selectors`. |
| 487 | |
| 488 | ``` |
| 489 | summary.nodes[] — ni (index), mg (machine GUID), nd (node UUID), nm (hostname), st (status), sts (stats) |
| 490 | summary.contexts[] — id, is (instances count), ds (dimensions count), al (alerts), sts (stats) |
| 491 | summary.instances[] — id, nm (name), ni (node index), ds (dimensions count), al (alerts), sts (stats) |
| 492 | summary.dimensions[] — id, nm (name), ds (count), pri (priority), sts (stats) |
| 493 | summary.labels[] — id (label key), vl[] (label values with id and stats) |
| 494 | summary.alerts[] — nm (name), cl (clear count), wr (warning count), cr (critical count) |
| 495 | ``` |
| 496 | |
| 497 | Stats object (`sts`): `min`, `max`, `avg` (average), `arp` (anomaly rate %), `con` (contribution %). |
| 498 | |
| 499 | ItemsCount fields: `sl` (selected), `ex` (excluded), `qr` (query success), `fl` (query fail). |
| 500 | |
| 501 | ### view |
| 502 | |
| 503 | | Field | Description | |
| 504 | |-------|-------------| |
| 505 | | `title` | Chart title | |
| 506 | | `update_every` | Data collection interval (seconds) | |
| 507 | | `after` | Actual start timestamp of returned data | |
| 508 | | `before` | Actual end timestamp of returned data | |
| 509 | | `points` | Number of data points returned | |
| 510 | | `units` | Unit of measurement | |
| 511 | | `chart_type` | Default chart type (line, area, stacked) | |
| 512 | | `min` | Minimum value across all data | |
| 513 | | `max` | Maximum value across all data | |
| 514 | | `dimensions.grouped_by` | Array confirming the `group_by` used | |
| 515 | | `dimensions.ids` | Unique dimension IDs | |
| 516 | | `dimensions.names` | Human-readable dimension names (column headers) | |
| 517 | | `dimensions.units` | Units per dimension | |
| 518 | | `dimensions.priorities` | Display priority per dimension | |
| 519 | | `dimensions.aggregated` | Number of source metrics aggregated into each dimension | |
| 520 | | `dimensions.sts` | Stats arrays per dimension: `min[]`, `max[]`, `avg[]`, `arp[]`, `con[]` | |
| 521 | |
| 522 | ### result — The Time-Series Data |
| 523 | |
| 524 | ```json |
| 525 | { |
| 526 | "labels": ["time", "host1", "host2"], |
| 527 | "point": {"value": 0, "arp": 1, "pa": 2}, |
| 528 | "data": [ |
| 529 | [1700000060, [5.23, 0, 0], [3.15, 0, 0]], |
| 530 | [1700000120, [4.87, 0, 0], [2.91, 0, 0]] |
| 531 | ] |
| 532 | } |
| 533 | ``` |
| 534 | |
| 535 | - `result.labels` — column names. First is always `"time"`. Rest match `view.dimensions.names`. |
| 536 | - `result.point` — maps positions within each value array: `{"value": 0, "arp": 1, "pa": 2}` |
| 537 | - `result.data` — array of rows: `[timestamp, [col1_values], [col2_values], ...]` |
| 538 | |
| 539 | Each value array contains 3 elements: |
| 540 | - **Index 0 (`value`)**: The metric value |
| 541 | - **Index 1 (`arp`)**: Anomaly rate (0-100). Percentage of raw samples in this interval flagged as anomalous by ML |
| 542 | - **Index 2 (`pa`)**: Point annotations bitmap. Values can be combined (OR'd): |
| 543 | |
| 544 | | Bit | Value | Meaning | |
| 545 | |-----|-------|---------| |
| 546 | | (none) | `0` | Normal data point — no issues | |
| 547 | | bit 0 | `1` | **Empty** — no data was collected for this interval | |
| 548 | | bit 1 | `2` | **Reset** — a counter reset/overflow was detected | |
| 549 | | bit 2 | `4` | **Partial** — not all expected sources contributed to this point (e.g., in group-by queries, some series had no data) | |
| 550 | |
| 551 | Values combine: e.g., `5` = empty + partial, `6` = reset + partial. |
| 552 | |
| 553 | ### db |
| 554 | |
| 555 | | Field | Description | |
| 556 | |-------|-------------| |
| 557 | | `tiers` | Number of database tiers | |
| 558 | | `update_every` | Maximum update interval across nodes | |
| 559 | | `first_entry` | Earliest data timestamp | |
| 560 | | `last_entry` | Latest data timestamp | |
| 561 | | `per_tier[]` | Per-tier info: `tier`, `queries`, `points`, `update_every`, `first_entry`, `last_entry` | |
| 562 | | `units` | Database units | |
| 563 | | `dimensions.ids` | Database dimension IDs | |
| 564 | | `dimensions.units` | Database dimension units | |
| 565 | | `dimensions.sts` | Database-level stats | |
| 566 | |
| 567 | ### timings |
| 568 | |
| 569 | | Field | Description | |
| 570 | |-------|-------------| |
| 571 | | `total_ms` | Total query time | |
| 572 | | `routing_ms` | Time to route to agents | |
| 573 | | `prep_ms` | Preparation time (per agent) | |
| 574 | | `query_ms` | Query execution time (per agent) | |
| 575 | | `output_ms` | Output formatting time (per agent) | |
| 576 | | `node_max_ms` | Slowest node response time | |
| 577 | | `cloud_ms` | Cloud processing time | |
| 578 | |
| 579 | --- |
| 580 | |
| 581 | ## How Users Find Metric Names in the UI |
| 582 | |
| 583 | 1. **Context names** (for `scope.contexts`): |
| 584 | - The context is shown next to the chart title (e.g., `system.cpu`, `disk.space`). You can click it to copy it. |
| 585 | - Use the **Metrics** tab in the dashboard to browse all available contexts |
| 586 | - Use the `/contexts` endpoint with `scope.contexts: ["pattern*"]` |
| 587 | |
| 588 | 2. **Dimension names** (for `scope.dimensions`): |
| 589 | - Visible in the chart legend (e.g., `user`, `system`, `iowait` for CPU) |
| 590 | - Query with `group_by: ["dimension"]` to see all dimension names in `view.dimensions.names` |
| 591 | |
| 592 | 3. **Node hostnames and UUIDs** (for `scope.nodes`): |
| 593 | - The **Nodes** tab lists hostnames |
| 594 | - Use `/nodes` endpoint to get UUIDs (the `nd` field) |
| 595 | |
| 596 | 4. **Labels** (for `scope.labels` and `group_by_label`): |
| 597 | - Click the labels drop-down on a chart, to see all label keys and values |
| 598 | - Labels like `mount_point`, `filesystem`, `interface` appear in the list |
| 599 | - Query with `group_by: ["selected"]` and check `summary.labels` in the response to discover available label keys and values for a context |
| 600 | |
| 601 | --- |
| 602 | |
| 603 | ## Practical Examples |
| 604 | |
| 605 | All examples use this pattern — users replace the 3 variables at the top: |
| 606 | |
| 607 | ```bash |
| 608 | TOKEN="YOUR_API_TOKEN" |
| 609 | SPACE="YOUR_SPACE_ID" |
| 610 | ROOM="YOUR_ROOM_ID" |
| 611 | ``` |
| 612 | |
| 613 | ### Example 1: Total CPU Across All Nodes (Last 10 Minutes) |
| 614 | |
| 615 | ```bash |
| 616 | TOKEN="YOUR_API_TOKEN" |
| 617 | SPACE="YOUR_SPACE_ID" |
| 618 | ROOM="YOUR_ROOM_ID" |
| 619 | |
| 620 | read -r -d '' PAYLOAD <<'EOF' |
| 621 | { |
| 622 | "scope": {"contexts": ["system.cpu"]}, |
| 623 | "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]}, |
| 624 | "window": {"after": -600, "before": 0, "points": 5}, |
| 625 | "aggregations": { |
| 626 | "metrics": [{"group_by": ["selected"], "aggregation": "sum"}], |
| 627 | "time": {"time_group": "average"} |
| 628 | }, |
| 629 | "format": "json2", |
| 630 | "options": ["jsonwrap", "minify", "unaligned"], |
| 631 | "timeout": 30000 |
| 632 | } |
| 633 | EOF |
| 634 | |
| 635 | curl -s -X POST \ |
| 636 | -H 'Content-Type: application/json' \ |
| 637 | -H "Authorization: Bearer $TOKEN" \ |
| 638 | "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \ |
| 639 | -d "$PAYLOAD" |
| 640 | ``` |
| 641 | |
| 642 | Result: Single column `selected` with total CPU % (sum of all dimensions across all nodes) at 5 time points. |
| 643 | |
| 644 | ### Example 2: CPU Breakdown by Dimension |
| 645 | |
| 646 | ```bash |
| 647 | TOKEN="YOUR_API_TOKEN" |
| 648 | SPACE="YOUR_SPACE_ID" |
| 649 | ROOM="YOUR_ROOM_ID" |
| 650 | |
| 651 | read -r -d '' PAYLOAD <<'EOF' |
| 652 | { |
| 653 | "scope": {"contexts": ["system.cpu"]}, |
| 654 | "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]}, |
| 655 | "window": {"after": -600, "before": 0, "points": 5}, |
| 656 | "aggregations": { |
| 657 | "metrics": [{"group_by": ["dimension"], "aggregation": "sum"}], |
| 658 | "time": {"time_group": "average"} |
| 659 | }, |
| 660 | "format": "json2", |
| 661 | "options": ["jsonwrap", "minify", "unaligned"], |
| 662 | "timeout": 30000 |
| 663 | } |
| 664 | EOF |
| 665 | |
| 666 | curl -s -X POST \ |
| 667 | -H 'Content-Type: application/json' \ |
| 668 | -H "Authorization: Bearer $TOKEN" \ |
| 669 | "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \ |
| 670 | -d "$PAYLOAD" |
| 671 | ``` |
| 672 | |
| 673 | Result: One column per dimension (`user`, `system`, `iowait`, `irq`, `softirq`, `steal`, `guest`, `nice`). Values are summed across all nodes. |
| 674 | |
| 675 | ### Example 3: Compare CPU Per Node |
| 676 | |
| 677 | ```bash |
| 678 | TOKEN="YOUR_API_TOKEN" |
| 679 | SPACE="YOUR_SPACE_ID" |
| 680 | ROOM="YOUR_ROOM_ID" |
| 681 | |
| 682 | read -r -d '' PAYLOAD <<'EOF' |
| 683 | { |
| 684 | "scope": {"contexts": ["system.cpu"]}, |
| 685 | "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]}, |
| 686 | "window": {"after": -600, "before": 0, "points": 5}, |
| 687 | "aggregations": { |
| 688 | "metrics": [{"group_by": ["node"], "aggregation": "sum"}], |
| 689 | "time": {"time_group": "average"} |
| 690 | }, |
| 691 | "format": "json2", |
| 692 | "options": ["jsonwrap", "minify", "unaligned"], |
| 693 | "timeout": 30000 |
| 694 | } |
| 695 | EOF |
| 696 | |
| 697 | curl -s -X POST \ |
| 698 | -H 'Content-Type: application/json' \ |
| 699 | -H "Authorization: Bearer $TOKEN" \ |
| 700 | "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \ |
| 701 | -d "$PAYLOAD" |
| 702 | ``` |
| 703 | |
| 704 | Result: One column per node hostname. Values are total CPU % per node. Column names in `view.dimensions.names`. |
| 705 | |
| 706 | ### Example 4: Peak CPU Per Node Over Last Hour |
| 707 | |
| 708 | ```bash |
| 709 | TOKEN="YOUR_API_TOKEN" |
| 710 | SPACE="YOUR_SPACE_ID" |
| 711 | ROOM="YOUR_ROOM_ID" |
| 712 | |
| 713 | read -r -d '' PAYLOAD <<'EOF' |
| 714 | { |
| 715 | "scope": {"contexts": ["system.cpu"]}, |
| 716 | "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]}, |
| 717 | "window": {"after": -3600, "before": 0, "points": 6}, |
| 718 | "aggregations": { |
| 719 | "metrics": [{"group_by": ["node"], "aggregation": "max"}], |
| 720 | "time": {"time_group": "max"} |
| 721 | }, |
| 722 | "format": "json2", |
| 723 | "options": ["jsonwrap", "minify", "unaligned"], |
| 724 | "timeout": 30000 |
| 725 | } |
| 726 | EOF |
| 727 | |
| 728 | curl -s -X POST \ |
| 729 | -H 'Content-Type: application/json' \ |
| 730 | -H "Authorization: Bearer $TOKEN" \ |
| 731 | "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \ |
| 732 | -d "$PAYLOAD" |
| 733 | ``` |
| 734 | |
| 735 | Result: 6 points (10-min intervals). Each value is the **peak** CPU for that node in that interval. |
| 736 | |
| 737 | ### Example 5: Disk Space Grouped by Filesystem Type |
| 738 | |
| 739 | ```bash |
| 740 | TOKEN="YOUR_API_TOKEN" |
| 741 | SPACE="YOUR_SPACE_ID" |
| 742 | ROOM="YOUR_ROOM_ID" |
| 743 | |
| 744 | read -r -d '' PAYLOAD <<'EOF' |
| 745 | { |
| 746 | "scope": {"contexts": ["disk.space"]}, |
| 747 | "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]}, |
| 748 | "window": {"after": -600, "before": 0, "points": 5}, |
| 749 | "aggregations": { |
| 750 | "metrics": [{"group_by": ["label"], "group_by_label": ["filesystem"], "aggregation": "sum"}], |
| 751 | "time": {"time_group": "average"} |
| 752 | }, |
| 753 | "format": "json2", |
| 754 | "options": ["jsonwrap", "minify", "unaligned"], |
| 755 | "timeout": 30000 |
| 756 | } |
| 757 | EOF |
| 758 | |
| 759 | curl -s -X POST \ |
| 760 | -H 'Content-Type: application/json' \ |
| 761 | -H "Authorization: Bearer $TOKEN" \ |
| 762 | "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \ |
| 763 | -d "$PAYLOAD" |
| 764 | ``` |
| 765 | |
| 766 | Result: One column per filesystem type (ext4, btrfs, tmpfs, etc.). Values are total disk space summed across all nodes. |
| 767 | |
| 768 | ### Example 6: Filter by Specific Nodes (UUIDs) |
| 769 | |
| 770 | ```bash |
| 771 | TOKEN="YOUR_API_TOKEN" |
| 772 | SPACE="YOUR_SPACE_ID" |
| 773 | ROOM="YOUR_ROOM_ID" |
| 774 | |
| 775 | read -r -d '' PAYLOAD <<'EOF' |
| 776 | { |
| 777 | "scope": {"contexts": ["system.cpu"], "nodes": ["NODE_UUID_1", "NODE_UUID_2"]}, |
| 778 | "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]}, |
| 779 | "window": {"after": -600, "before": 0, "points": 5}, |
| 780 | "aggregations": { |
| 781 | "metrics": [{"group_by": ["node"], "aggregation": "sum"}], |
| 782 | "time": {"time_group": "average"} |
| 783 | }, |
| 784 | "format": "json2", |
| 785 | "options": ["jsonwrap", "minify", "unaligned"], |
| 786 | "timeout": 30000 |
| 787 | } |
| 788 | EOF |
| 789 | |
| 790 | curl -s -X POST \ |
| 791 | -H 'Content-Type: application/json' \ |
| 792 | -H "Authorization: Bearer $TOKEN" \ |
| 793 | "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \ |
| 794 | -d "$PAYLOAD" |
| 795 | ``` |
| 796 | |
| 797 | Result: Data and metadata scoped to only those 2 nodes. First call `/nodes` to get UUIDs (the `nd` field). |
| 798 | |
| 799 | ### Example 7: Filter by Labels |
| 800 | |
| 801 | ```bash |
| 802 | TOKEN="YOUR_API_TOKEN" |
| 803 | SPACE="YOUR_SPACE_ID" |
| 804 | ROOM="YOUR_ROOM_ID" |
| 805 | |
| 806 | read -r -d '' PAYLOAD <<'EOF' |
| 807 | { |
| 808 | "scope": {"contexts": ["disk.space"], "labels": ["mount_point:/", "filesystem:ext4"]}, |
| 809 | "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]}, |
| 810 | "window": {"after": -600, "before": 0, "points": 5}, |
| 811 | "aggregations": { |
| 812 | "metrics": [{"group_by": ["selected"], "aggregation": "sum"}], |
| 813 | "time": {"time_group": "average"} |
| 814 | }, |
| 815 | "format": "json2", |
| 816 | "options": ["jsonwrap", "minify", "unaligned"], |
| 817 | "timeout": 30000 |
| 818 | } |
| 819 | EOF |
| 820 | |
| 821 | curl -s -X POST \ |
| 822 | -H 'Content-Type: application/json' \ |
| 823 | -H "Authorization: Bearer $TOKEN" \ |
| 824 | "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \ |
| 825 | -d "$PAYLOAD" |
| 826 | ``` |
| 827 | |
| 828 | Result: Only ext4 root mount points. Multiple labels with different keys are AND-combined. |
| 829 | |
| 830 | ### Example 8: Filter Specific Dimensions |
| 831 | |
| 832 | ```bash |
| 833 | TOKEN="YOUR_API_TOKEN" |
| 834 | SPACE="YOUR_SPACE_ID" |
| 835 | ROOM="YOUR_ROOM_ID" |
| 836 | |
| 837 | read -r -d '' PAYLOAD <<'EOF' |
| 838 | { |
| 839 | "scope": {"contexts": ["system.cpu"], "dimensions": ["user", "system"]}, |
| 840 | "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]}, |
| 841 | "window": {"after": -600, "before": 0, "points": 5}, |
| 842 | "aggregations": { |
| 843 | "metrics": [{"group_by": ["dimension"], "aggregation": "sum"}], |
| 844 | "time": {"time_group": "average"} |
| 845 | }, |
| 846 | "format": "json2", |
| 847 | "options": ["jsonwrap", "minify", "unaligned"], |
| 848 | "timeout": 30000 |
| 849 | } |
| 850 | EOF |
| 851 | |
| 852 | curl -s -X POST \ |
| 853 | -H 'Content-Type: application/json' \ |
| 854 | -H "Authorization: Bearer $TOKEN" \ |
| 855 | "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \ |
| 856 | -d "$PAYLOAD" |
| 857 | ``` |
| 858 | |
| 859 | Result: Only `user` and `system` CPU dimensions. |
| 860 | |
| 861 | ### Example 9: Discover Labels for a Context |
| 862 | |
| 863 | ```bash |
| 864 | TOKEN="YOUR_API_TOKEN" |
| 865 | SPACE="YOUR_SPACE_ID" |
| 866 | ROOM="YOUR_ROOM_ID" |
| 867 | |
| 868 | read -r -d '' PAYLOAD <<'EOF' |
| 869 | { |
| 870 | "scope": {"contexts": ["disk.space"]}, |
| 871 | "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]}, |
| 872 | "window": {"after": -600, "before": 0, "points": 1}, |
| 873 | "aggregations": { |
| 874 | "metrics": [{"group_by": ["selected"], "aggregation": "sum"}], |
| 875 | "time": {"time_group": "average"} |
| 876 | }, |
| 877 | "format": "json2", |
| 878 | "options": ["jsonwrap", "minify", "unaligned"], |
| 879 | "timeout": 30000 |
| 880 | } |
| 881 | EOF |
| 882 | |
| 883 | curl -s -X POST \ |
| 884 | -H 'Content-Type: application/json' \ |
| 885 | -H "Authorization: Bearer $TOKEN" \ |
| 886 | "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \ |
| 887 | -d "$PAYLOAD" |
| 888 | ``` |
| 889 | |
| 890 | Then inspect `summary.labels` in the response: |
| 891 | |
| 892 | ```json |
| 893 | "summary": { |
| 894 | "labels": [ |
| 895 | {"id": "filesystem", "vl": [{"id": "ext4"}, {"id": "btrfs"}, {"id": "tmpfs"}]}, |
| 896 | {"id": "mount_point", "vl": [{"id": "/"}, {"id": "/boot"}, {"id": "/home"}]} |
| 897 | ] |
| 898 | } |
| 899 | ``` |
| 900 | |
| 901 | --- |
| 902 | |
| 903 | ## Known Limitations |
| 904 | |
| 905 | 1. **`scope.contexts` MUST always be set** — without it, the response includes metadata for every metric in the room (hundreds of contexts, thousands of instances). This causes multi-megabyte responses. |
| 906 | 2. **`scope.nodes` accepts only node UUIDs** — use `selectors.nodes` with hostname patterns instead (simpler). Only use `scope.nodes` when you need tight metadata scoping. |
| 907 | 3. **Only `json2` format** is supported by the Cloud API. Other formats (csv, ssv, etc.) are not reliably supported through the Cloud proxy. |
| 908 | 4. **Max ~500 data points** per query. The Cloud clamps requests to 500 before forwarding to agents; actual returned count may vary slightly due to time alignment. |
| 909 | 5. **Default timeout is 10 seconds** (10000ms). Increase for large/slow queries. |
| 910 | 6. **Stale nodes** appear in `/nodes` but return no data. Check `state` field. |
| 911 | 7. **Always use `unaligned` option** for API queries — without it, time intervals snap to wall-clock boundaries, which is confusing for programmatic use. |
| 912 | |
| 913 | --- |
| 914 | |
| 915 | ## Troubleshooting / FAQ |
| 916 | |
| 917 | **Q: My query returns empty data — no error, but no results either.** |
| 918 | A: The API returns empty responses when nothing matches your filters. This is by design — it does not return an error. Check: (1) Is `scope.contexts` set to a valid context name? Typos like `cpu.system` instead of `system.cpu` silently return nothing. (2) Are your `scope.labels` or `scope.dimensions` correct? (3) If using `scope.nodes`, are the UUIDs valid? Use `/contexts` to verify context names and `/nodes` to verify UUIDs. |
| 919 | |
| 920 | **Q: The response is huge (megabytes) and slow.** |
| 921 | A: You are missing `scope.contexts`. Without it, the scope defaults to the entire room — every context, instance, dimension, and label across all nodes. Always set `scope.contexts` to only the context(s) you need (e.g., `["system.cpu"]`). |
| 922 | |
| 923 | **Q: The time range in the response doesn't match what I requested.** |
| 924 | A: Add `"unaligned"` to the `options` array. Without it, time intervals are aligned to wall-clock boundaries based on the query period. For example, a 1-hour query might snap to 14:00–15:00 instead of 14:07–15:07. The `unaligned` option gives you the exact time range you asked for. |
| 925 | |
| 926 | **Q: I'm using hostnames in `scope.nodes` and getting no data.** |
| 927 | A: `scope.nodes` only accepts node UUIDs (the `nd` field from `/nodes`). Hostnames, patterns, and machine GUIDs do not work there. Use `selectors.nodes` instead — it accepts hostname patterns (e.g., `["web*", "prod-*"]`) and is the preferred way to filter by node. |
| 928 | |
| 929 | **Q: I requested CSV format but got an error or garbled output.** |
| 930 | A: Only `json2` format works through the Cloud API. The Cloud proxy cannot aggregate CSV responses from multiple agents. Always use `"format": "json2"`. |
| 931 | |
| 932 | **Q: How do I find the context name for a metric I see on the dashboard?** |
| 933 | A: The context name is shown next to the chart title on every Netdata chart (e.g., `system.cpu`, `disk.space`, `net.net`). Click it to copy it to the clipboard. You can also use the `/contexts` endpoint with a pattern like `["system.*"]` to browse available contexts. |
| 934 | |
| 935 | **Q: The anomaly rate (`arp`) is always 0 — is anomaly detection working?** |
| 936 | A: An `arp` of 0 means either no anomalies were detected (normal for healthy systems) or ML-based anomaly detection is disabled on the agent. Anomaly detection runs on every metric at collection time using ML (k-means clustering). Non-zero values indicate the percentage of raw samples in the interval that were flagged as anomalous. If `arp` is 0 across all metrics and all time ranges, the agent may have ML disabled. |
| 937 | |
| 938 | **Q: `countif` or `percentile` time_group returns unexpected values.** |
| 939 | A: These functions require raw per-second data. Add `"tier": 0` to the `window` object to force the use of the non-aggregated storage tier. Without it, the query may use a pre-aggregated tier (per-minute or per-hour) where these functions cannot work correctly. |
| 940 | |
| 941 | **Q: I see non-zero `pa` values in the data — what do they mean?** |
| 942 | A: `pa` is a point annotations bitmap: `1` = empty (no data collected), `2` = counter reset/overflow detected, `4` = partial (not all sources contributed in a group-by query). Values combine: e.g., `5` = empty + partial. Non-zero `pa` values are common at query boundaries and during agent restarts. |
| 943 | |
| 944 | **Q: How do I filter nodes by hostname without looking up UUIDs?** |
| 945 | A: Use `selectors.nodes` with hostname patterns: `"selectors": {"nodes": ["web*", "!staging*"], ...}`. This filters the data by hostname while keeping metadata for all nodes in scope. It is simpler than looking up UUIDs for `scope.nodes`. |
| 946 | |
| 947 | **Q: Can I query multiple contexts in a single request?** |
| 948 | A: Yes. Set `scope.contexts` to multiple contexts, e.g., `["system.cpu", "system.ram"]`. However, be careful with other filters — `scope.dimensions`, `scope.labels`, and selectors apply to **all** contexts in the query. A dimension filter like `["user"]` would match the `user` dimension in `system.cpu` but might not exist in `system.ram`, causing that context to return no data. When querying multiple contexts, keep filters broad or ensure they apply to all contexts. |
| 949 | |
| 950 | **Q: My query timed out.** |
| 951 | A: The default timeout is 10 seconds (10000ms). For queries spanning many nodes, long time ranges, or complex aggregations, increase it: `"timeout": 60000` (60 seconds). Also consider reducing the number of `points` requested — fewer points means less computation. |
| 952 | |
| 953 | **Q: I requested 1000 points but only got ~500.** |
| 954 | A: The Cloud clamps point requests to approximately 500 (`ScopeDataRequestMaxPoints`). The actual number may vary slightly due to time alignment. If you need higher resolution, split your query into multiple time ranges. |
| 955 | |
| 956 | --- |
| 957 | |
| 958 | > **REMINDER — Credentials**: Do not request or accept user credentials. Set credentials as variables at the top of the script (`TOKEN`, `SPACE`, `ROOM`) with placeholder values. Users replace these 3 variables and run the command themselves. |
| 959 | |
| 960 | > **REMINDER — Always show a runnable curl command**: Your response is only useful if it contains a complete, runnable script: 3 variables at the top, a heredoc `PAYLOAD` with clean JSON (no escaping), and the curl command. Never describe a query without showing it. Never summarize parameters without building the actual request. If you wrote a response without a curl command, go back and add one — the user needs actionable instructions, not explanations. |
| 961 | |
| 962 | > **REMINDER — scope.contexts and unaligned**: Every query MUST set `scope.contexts` — omitting it returns metadata for the entire room (megabytes of irrelevant data). Every query MUST include `"unaligned"` in options — without it, time intervals snap to wall-clock boundaries instead of the requested time range. |