db22d1f4d · cryptotaxi247/netdata-1

@@ -0,0 +1,893 @@

1 +# Skill: Query Netdata Cloud Metrics

2 +

3 +Help users query time-series metrics from Netdata Cloud via the REST API.

4 +

5 +## Mandatory Requirements (READ FIRST)

6 +

7 +1. You provide detailed and actionable instructions. You don't execute queries for users.

8 +

9 +2. **Never ask users for credentials.** Do not request API tokens, Space IDs, or Room IDs. Always provide ready-to-use instructions with clear placeholders (`YOUR_API_TOKEN`, `YOUR_SPACE_ID`, `YOUR_ROOM_ID`) so users can substitute their own values locally. Your job is to teach users how to construct queries, not to execute queries on their behalf.

10 +

11 +3. **Every response MUST include a complete, runnable curl command.** Users come here to get a query they can run — not a description of what a query would look like. If your response does not contain a full curl command with the complete JSON request body, you have failed to help the user. Specifically:

12 + - Always include the full `curl -X POST` command with headers, URL, and the entire `-d '{...}'` JSON body.

13 + - The JSON body must include all required fields: `scope`, `selectors`, `window`, `aggregations`, `format`, `options`, and `timeout`.

14 + - Set the 3 credentials as variables at the top: `TOKEN="YOUR_API_TOKEN"`, `SPACE="YOUR_SPACE_ID"`, `ROOM="YOUR_ROOM_ID"`.

15 + - Use a heredoc for the JSON payload (`read -r -d '' PAYLOAD <<'EOF' ... EOF`) so no escaping is needed.

16 + - The user must be able to copy your command, replace the 3 variables at the top, and run it immediately in their terminal.

17 + - A response that describes parameters or explains concepts without providing the actual runnable command is incomplete and unhelpful.

18 + - Even for simple questions, always provide the curl command. When in doubt, show the command.

19 +

20 +---

21 +

22 +## Prerequisites

23 +

24 +Three things are needed:

25 +

26 +### 1. API Token

27 +

28 +1. Login to [app.netdata.cloud](https://app.netdata.cloud)

29 +2. Click user icon (lower-left corner, tooltip shows your name)

30 +3. Select **User Settings**

31 +4. In the modal, select the **API Tokens** tab

32 +5. Click the **[+]** button (top-left)

33 +6. Select a scope, enter a description, click **Create**

34 +7. **Copy the token immediately** — it will not be shown again

35 +

36 +Relevant scopes: `scope:all` (full access), `scope:grafana-plugin` (data endpoints).

37 +

38 +### 2. Space ID

39 +

40 +1. In the dashboard left side, at the spaces list, click the **gear icon** below the spaces list (tooltip: "Space Settings")

41 +2. In the **Info** tab, copy the **Space Id**

42 +

43 +### 3. Room ID

44 +

45 +1. In the same Space Settings, go to the **Rooms** tab

46 +2. Find the room, click the **>** icon at the right of the room row (tooltip: "Room Settings")

47 +3. In the **Room** tab, copy the **Room Id**

48 +

49 +---

50 +

51 +## API Endpoints

52 +

53 +Base URL: `https://app.netdata.cloud`

54 +Swagger online: https://app.netdata.cloud/api/docs/

55 +

56 +All endpoints use **POST** with a JSON body and require:

57 +

58 +```

59 +Authorization: Bearer YOUR_API_TOKEN

60 +Content-Type: application/json

61 +```

62 +

63 +| Endpoint | Purpose |

64 +|----------|---------|

65 +| `/api/v3/spaces/{spaceID}/rooms/{roomID}/data` | Query time-series data |

66 +| `/api/v3/spaces/{spaceID}/rooms/{roomID}/nodes` | List nodes in the room |

67 +| `/api/v3/spaces/{spaceID}/rooms/{roomID}/contexts` | List available metric contexts |

68 +

69 +---

70 +

71 +## Discover Nodes

72 +

73 +**Endpoint:** POST `/api/v3/spaces/{spaceID}/rooms/{roomID}/nodes`

74 +**Body:** `{}`

75 +

76 +Response fields per node:

77 +

78 +| JSON field | Description |

79 +|------------|-------------|

80 +| `nd` | **Node UUID** — required for `scope.nodes` in data queries |

81 +| `nm` | Hostname |

82 +| `mg` | Machine GUID |

83 +| `state` | `reachable` (live) or `stale` (disconnected) |

84 +| `v` | Agent version |

85 +| `labels` | All node labels as key-value pairs |

86 +| `hw` | Hardware: `cpus`, `memory`, `disk_space`, `architecture` |

87 +| `os` | OS: `nm` (name), `v` (version), `kernel` |

88 +| `health` | Alert summary: `status`, `alerts.warning`, `alerts.critical` |

89 +| `capabilities` | Supported features: `ml`, `funcs`, `health`, etc. |

90 +

91 +Example:

92 +

93 +```bash

94 +TOKEN="YOUR_API_TOKEN"

95 +SPACE="YOUR_SPACE_ID"

96 +ROOM="YOUR_ROOM_ID"

97 +

98 +curl -s -X POST \

99 + -H 'Content-Type: application/json' \

100 + -H "Authorization: Bearer $TOKEN" \

101 + "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/nodes" \

102 + -d '{}'

103 +```

104 +

105 +---

106 +

107 +## Discover Contexts

108 +

109 +Contexts are metric types (e.g., `system.cpu`, `disk.space`, `net.net`).

110 +

111 +**Endpoint:** POST `/api/v3/spaces/{spaceID}/rooms/{roomID}/contexts`

112 +

113 +```json

114 +{

115 + "scope": { "contexts": ["system.*"] },

116 + "selectors": { "nodes": ["*"], "contexts": ["*"] }

117 +}

118 +```

119 +

120 +`scope.contexts` supports patterns: `system.*`, `disk.*`, `*cpu*`.

121 +

122 +---

123 +

124 +## Query Metric Data

125 +

126 +**Endpoint:** POST `/api/v3/spaces/{spaceID}/rooms/{roomID}/data`

127 +

128 +### Full Request Body Structure

129 +

130 +```json

131 +{

132 + "scope": {

133 + "nodes": [],

134 + "contexts": [],

135 + "instances": [],

136 + "dimensions": [],

137 + "labels": []

138 + },

139 + "selectors": {

140 + "nodes": ["*"],

141 + "contexts": ["*"],

142 + "instances": ["*"],

143 + "dimensions": ["*"],

144 + "labels": ["*"],

145 + "alerts": ["*"]

146 + },

147 + "window": {

148 + "after": 0,

149 + "before": 0,

150 + "points": 0,

151 + "duration": 0,

152 + "tier": null,

153 + "baseline": null

154 + },

155 + "aggregations": {

156 + "metrics": [

157 + {

158 + "group_by": [],

159 + "group_by_label": [],

160 + "aggregation": "avg"

161 + }

162 + ],

163 + "time": {

164 + "time_group": "average",

165 + "time_group_options": null,

166 + "time_resampling": null

167 + }

168 + },

169 + "format": "json2",

170 + "options": [],

171 + "timeout": 10000,

172 + "limit": null

173 +}

174 +```

175 +

176 +---

177 +

178 +### scope — Define the Data Universe

179 +

180 +Scope controls **both data and metadata** in the response. Use scope fields for filtering so that the response metadata is focused on what you asked for.

181 +

183 +|-------|------|---------|---------------------|

189 +

190 +Multiple entries in the same field are OR-combined. Multiple `labels` entries with different keys are AND-combined.

191 +

192 +**LIMITATION: `scope.nodes` accepts only node UUIDs.** Hostnames, hostname patterns, machine GUIDs, and wildcards do not work via the Cloud API. To filter by node:

193 +1. Call `/nodes` to get UUIDs (the `nd` field)

194 +2. Use those UUIDs in `scope.nodes`

195 +3. Omit `scope.nodes` entirely to include all nodes

196 +

197 +---

198 +

199 +### selectors — Further Filter Data Within the Scope

200 +

201 +Selectors filter **data only** — response metadata still reflects the full scope. For programmatic API queries, use `scope` for filtering and set all selectors to `["*"]`.

202 +

203 +Selectors exist for the Netdata dashboard, which needs full metadata to show context ("the whole") while displaying a filtered subset.

204 +

206 +|-------|------|----------------|----------|

213 +

214 +**Note:** `selectors.nodes` is the only way to filter by hostname pattern via the Cloud API. Use it when you cannot look up UUIDs first, but be aware metadata will include all nodes in scope.

215 +

216 +---

217 +

218 +### window — Time Range

219 +

221 +|-------|------|-------------|---------|

222 +| `after` | `int` | Start time. Negative = relative seconds from `before` (max -94608000 = 3 years). Positive = Unix epoch. | `-600` |

223 +| `before` | `int` | End time. Negative = relative seconds from now (max -94608000). Positive = Unix epoch. | `0` (now) |

224 +| `points` | `int` | Number of data points to return. `0` or omitted = all available points. | `0` |

225 +| `duration` | `int` | Alternative to after/before. Duration in seconds. | `0` |

228 +

229 +Max points requested: approximately **500** (`ScopeDataRequestMaxPoints`). The Cloud clamps the request to 500 before forwarding to agents, but the actual number returned may vary slightly due to time alignment.

230 +

231 +The time range is divided into `points` equal intervals. Each interval is aggregated using the `time_group` function.

232 +

233 +---

234 +

235 +### How the Query Pipeline Works

236 +

237 +The query engine is a pipeline with two aggregation stages:

238 +

239 +1. **Identify time-series** matching the `scope` and `selectors`

240 +2. **Set up the output time-series** based on `group_by` (e.g., 2 groups for label values A and B)

241 +3. **For each matched time-series:**

242 + - **Stage 1 — Time aggregation** (`time_group`): Aggregate raw samples within each time interval into `points` data points (e.g., average 86400 per-second samples into 100 points)

243 + - **Stage 2 — Metric aggregation** (`aggregation`): Add the time-aggregated points into the appropriate output time-series using the aggregation function (e.g., SUM into group A or B)

244 +4. **Present** the grouped, aggregated result

245 +

246 +**Key insight**: `time_group` reduces samples within each time-series. `aggregation` combines multiple time-series into groups. They operate in sequence — metric aggregation works on already time-aggregated data.

247 +

248 +#### Example: 1000 containers, group by label `namespace` (2 values: A and B), 100 points over 1 day

249 +

250 +1. Output setup: 2 time-series needed (A and B), each with 100 points

251 +2. For each of the 1000 container time-series:

252 + - Time-aggregate 86400 seconds into 100 points using `time_group` (e.g., `average`)

253 + - Add those 100 points into either A or B using `aggregation` (e.g., `sum`)

254 +3. Result: 2 columns (A, B) × 100 rows

255 +

256 +#### Choosing time_group Based on What the User Wants

257 +

258 +| User intent | time_group | Why |

259 +|-------------|-----------|-----|

260 +| Average resource consumption (rate metrics: CPU, I/O, bandwidth) | `average` | Rate metrics represent per-second rates; averaging preserves the rate |

261 +| Average resource consumption (gauge metrics: memory, disk space, connections) | `average` or `max` | Gauges represent current state; max shows peak usage |

262 +| Find spikes or peaks (any metric type) | `max` | Captures the highest value within each interval |

263 +| Total volume transferred (counters: bytes, packets) | `sum` | Sums the actual volume |

264 +| Count events matching a condition | `countif` | Counts samples matching a threshold |

265 +

266 +#### Choosing aggregation Based on How to Combine Series

267 +

268 +| User intent | aggregation | Why |

269 +|-------------|------------|-----|

270 +| Total across all series (e.g., total CPU across all containers) | `sum` | Adds up all contributions |

271 +| Average across series | `avg` | Mean of the group |

272 +| Worst case across series | `max` | Highest value in the group |

273 +| Best case across series | `min` | Lowest value in the group |

274 +

275 +#### Mapping User Questions to Parameters

276 +

277 +**"Find a CPU spike over the last week across all my containers"**

278 +→ `time_group: "max"`, `aggregation: "sum"` (sum user+system), `group_by: ["instance"]`

279 +

280 +**"Which namespace consumed most CPU over the last week?"**

281 +→ `time_group: "average"` (per-second rate) or `"sum"` (total), `aggregation: "sum"`, `group_by: ["label"]`, `group_by_label: ["namespace"]`

282 +

283 +**"Peak memory usage per node over the last 24 hours"**

284 +→ `time_group: "max"` (gauge metric, want peak), `aggregation: "sum"`, `group_by: ["node"]`

285 +

286 +#### Research the Context Before Answering

287 +

288 +Before constructing a query for a user, you should understand the metric context they are asking about — its dimensions, labels, and whether it represents rates (`incremental`) or gauges (`absolute`). Search for the context name (e.g., `cgroup.cpu`, `disk.space`, `nginx.connections`) in the Netdata source code to find its `metadata.yaml`, which defines dimensions, units, chart type, and available labels. This ensures you choose the correct `time_group` and `aggregation` for their use case.

289 +

290 +---

291 +

292 +### aggregations.time — Time Aggregation

293 +

294 +Controls how raw data points within each time interval are combined into one value per series.

295 +

297 +|-------|------|-------------|---------|

301 +

302 +#### time_group values

303 +

304 +| Value | Aliases | Description |

305 +|-------|---------|-------------|

306 +| `average` | `avg` | Mean value **(default)** |

307 +| `min` | | Minimum value |

308 +| `max` | | Maximum value |

309 +| `sum` | | Sum of values |

310 +| `median` | | Median value |

311 +| `stddev` | | Standard deviation |

312 +| `cv` | | Coefficient of variation (stddev/mean) |

313 +| `ses` | | Single exponential smoothing |

314 +| `des` | | Double exponential smoothing |

315 +| `incremental-sum` | | Difference between last and first value in interval |

316 +| `countif` | | Count values matching condition. Set condition in `time_group_options`: `">0"`, `"=0"`, `"!=0"`, `"<=10"` |

317 +| `percentile` | | Percentile. Set percentile value in `time_group_options`: `"95"`, `"99"` |

318 +| `percentile25` | | 25th percentile (no options needed) |

319 +| `percentile50` | | 50th percentile |

320 +| `percentile75` | | 75th percentile |

321 +| `percentile80` | | 80th percentile |

322 +| `percentile90` | | 90th percentile |

323 +| `percentile95` | | 95th percentile |

324 +| `percentile97` | | 97th percentile |

325 +| `percentile98` | | 98th percentile |

326 +| `percentile99` | | 99th percentile |

327 +| `trimmed-mean` | | Mean after trimming outliers. Set trim % in `time_group_options` |

328 +| `trimmed-mean1` | | Trimmed mean, 1% trim |

329 +| `trimmed-mean2` | | Trimmed mean, 2% trim |

330 +| `trimmed-mean3` | | Trimmed mean, 3% trim |

331 +| `trimmed-mean5` | | Trimmed mean, 5% trim |

332 +| `trimmed-mean10` | | Trimmed mean, 10% trim |

333 +| `trimmed-mean15` | | Trimmed mean, 15% trim |

334 +| `trimmed-mean20` | | Trimmed mean, 20% trim |

335 +| `trimmed-mean25` | | Trimmed mean, 25% trim |

336 +| `trimmed-median` | | Median after trimming outliers. Set trim % in `time_group_options` |

337 +| `trimmed-median1` through `trimmed-median25` | | Same variants as trimmed-mean |

338 +

339 +IMPORTANT: when specifying any time_group except `min`, `max`, `avg`, `sum`, you MUST specify tier=0 to ensure a non-aggregated tier is used.

340 +

341 +#### time_group_options values

342 +

343 +| Used with | Value format | Example |

344 +|-----------|-------------|---------|

345 +| `countif` | Comparison operator + value | `">0"`, `"=0"`, `"!=0"`, `"<=100"` |

346 +| `percentile` | Percentile value (0-100) | `"95"`, `"99.5"` |

347 +| `trimmed-mean` | Trim percentage | `"5"`, `"10"` |

348 +| `trimmed-median` | Trim percentage | `"5"`, `"10"` |

349 +

350 +IMPORTANT: when specifying any time_group except `min`, `max`, `avg`, `sum`, you MUST specify tier=0 to ensure a non-aggregated tier is used.

351 +

352 +---

353 +

354 +### aggregations.metrics[] — Dimension Aggregation

355 +

356 +Controls how multiple time-series are combined. Each entry defines a grouping pass. At least one is required.

357 +

359 +|-------|------|-------------|---------|

363 +

364 +#### group_by values

365 +

366 +All values can be combined together **except** `selected` (if `selected` is present, all others are ignored).

367 +

368 +| Value | Result columns represent | Use case |

369 +|-------|------------------------|----------|

370 +| `selected` | Single column: all matched data combined into one series | Total/aggregate value across everything |

371 +| `dimension` | One column per unique dimension name | Break down by metric component (user/system/iowait for CPU) |

372 +| `node` | One column per node | Compare nodes side by side |

373 +| `instance` | One column per instance (`context@hostname`) | Compare instances across nodes |

374 +| `label` | One column per unique label value | Group by label (requires `group_by_label`) |

375 +| `context` | One column per context | Compare different metric types |

376 +| `units` | One column per unit type | Group by measurement unit |

377 +| `percentage-of-instance` | Percentages per dimension within each instance | Show proportions instead of absolutes |

378 +

379 +Combination example: `"group_by": ["node", "dimension"]` creates one column per node+dimension combination.

380 +

381 +#### aggregation values

382 +

383 +| Value | Aliases | Description |

384 +|-------|---------|-------------|

385 +| `avg` | `average` | Mean of grouped values **(default)** |

386 +| `sum` | | Sum of grouped values |

387 +| `min` | | Minimum of grouped values |

388 +| `max` | | Maximum of grouped values |

389 +| `median` | | Median of grouped values |

390 +| `percentage` | | Express as percentage of total |

391 +

392 +---

393 +

394 +### format

395 +

396 +Only `json2` is supported by Netdata Cloud.

397 +

398 +---

399 +

400 +### options

401 +

402 +Array of strings. Each option modifies the response behavior.

403 +

404 +| Option | Description |

405 +|--------|-------------|

406 +| `jsonwrap` | **Recommended.** Wraps the result with metadata (summary, view, db, timings) |

407 +| `minify` | **Recommended.** Minimizes JSON output size |

408 +| `nonzero` | Exclude dimensions that have only zero values |

409 +| `null2zero` | Replace null values with zero |

410 +| `abs` | Take absolute value of all data |

411 +| `absolute` | Same as `abs` |

412 +| `display-absolute` | Display absolute values |

413 +| `flip` | Flip the sign of values (multiply by -1) |

414 +| `reversed` | Reverse the order of data points (oldest last) |

415 +| `min2max` | Show the range (max - min) instead of the value |

416 +| `percentage` | Convert values to percentages |

417 +| `seconds` | Return timestamps as seconds |

418 +| `ms` | Return timestamps as milliseconds |

419 +| `milliseconds` | Same as `ms` |

420 +| `unaligned` | Do not align time intervals to round boundaries |

421 +| `match-ids` | Match dimensions by ID only (not name) |

422 +| `match-names` | Match dimensions by name only (not ID) |

423 +| `anomaly-bit` | Return anomaly rate instead of metric values |

424 +| `jw-anomaly-rates` | Include anomaly rates in jsonwrap metadata |

425 +| `details` | Include additional detail information |

426 +| `group-by-labels` | Include label information in view.dimensions for group-by results |

427 +| `natural-points` | Return natural data points (one per collection interval) |

428 +| `virtual-points` | Return virtual (interpolated) data points |

429 +| `selected-tier` | Force using the tier selected by the `tier` parameter |

430 +| `all-dimensions` | Include all dimensions, even those with no data |

431 +| `label-quotes` | Quote label values in output |

432 +| `objectrows` | Return data rows as objects instead of arrays |

433 +| `google_json` | Format compatible with Google Charts |

434 +| `raw` | Return raw data without trimming partial points |

435 +| `debug` | Include debug information |

436 +

437 +Recommended minimum: `["jsonwrap", "minify"]`

438 +

439 +---

440 +

441 +### timeout

442 +

443 +Query timeout in milliseconds. Default: `10000` (10 seconds). Set higher for queries spanning many nodes or long time ranges.

444 +

445 +### limit

446 +

447 +Optional integer. Limits the number of dimensions returned. Cannot be negative. Useful when querying high-cardinality contexts.

448 +

449 +---

450 +

451 +## Response Structure

452 +

453 +With `jsonwrap` option, the response contains:

454 +

455 +### Top-level fields

456 +

457 +| Field | Description |

458 +|-------|-------------|

459 +| `api` | API version (integer) |

460 +| `agents` | List of agents consulted |

461 +| `versions` | Hash values to detect database changes |

462 +| `summary` | Metadata about nodes, contexts, instances, dimensions, labels, alerts |

463 +| `totals` | Counts of selected/excluded/queried items |

464 +| `functions` | List of supported functions |

465 +| `db` | Database info (tiers, retention, update frequency) |

466 +| `view` | Presentation metadata (title, units, dimensions, time range) |

467 +| `result` | **The actual time-series data** |

468 +| `timings` | Query performance metrics |

469 +

470 +### summary

471 +

472 +Metadata determined by `scope`. Statistics within are influenced by `selectors`.

473 +

474 +```

475 +summary.nodes[] — ni (index), mg (machine GUID), nd (node UUID), nm (hostname), st (status), sts (stats)

476 +summary.contexts[] — id, is (instances count), ds (dimensions count), al (alerts), sts (stats)

477 +summary.instances[] — id, nm (name), ni (node index), ds (dimensions count), al (alerts), sts (stats)

478 +summary.dimensions[] — id, nm (name), ds (count), pri (priority), sts (stats)

479 +summary.labels[] — id (label key), vl[] (label values with id and stats)

480 +summary.alerts[] — nm (name), cl (clear count), wr (warning count), cr (critical count)

481 +```

482 +

483 +Stats object (`sts`): `min`, `max`, `avg` (average), `arp` (anomaly rate %), `con` (contribution %).

484 +

485 +ItemsCount fields: `sl` (selected), `ex` (excluded), `qr` (query success), `fl` (query fail).

486 +

487 +### view

488 +

489 +| Field | Description |

490 +|-------|-------------|

491 +| `title` | Chart title |

492 +| `update_every` | Data collection interval (seconds) |

493 +| `after` | Actual start timestamp of returned data |

494 +| `before` | Actual end timestamp of returned data |

495 +| `points` | Number of data points returned |

496 +| `units` | Unit of measurement |

497 +| `chart_type` | Default chart type (line, area, stacked) |

498 +| `min` | Minimum value across all data |

499 +| `max` | Maximum value across all data |

500 +| `dimensions.grouped_by` | Array confirming the `group_by` used |

501 +| `dimensions.ids` | Unique dimension IDs |

502 +| `dimensions.names` | Human-readable dimension names (column headers) |

503 +| `dimensions.units` | Units per dimension |

504 +| `dimensions.priorities` | Display priority per dimension |

505 +| `dimensions.aggregated` | Number of source metrics aggregated into each dimension |

506 +| `dimensions.sts` | Stats arrays per dimension: `min[]`, `max[]`, `avg[]`, `arp[]`, `con[]` |

507 +

508 +### result — The Time-Series Data

509 +

510 +```json

511 +{

512 + "labels": ["time", "host1", "host2"],

513 + "point": {"value": 0, "arp": 1, "pa": 2},

514 + "data": [

515 + [1700000060, [5.23, 0, 0], [3.15, 0, 0]],

516 + [1700000120, [4.87, 0, 0], [2.91, 0, 0]]

517 + ]

518 +}

519 +```

520 +

521 +- `result.labels` — column names. First is always `"time"`. Rest match `view.dimensions.names`.

522 +- `result.point` — maps positions within each value array: `{"value": 0, "arp": 1, "pa": 2}`

523 +- `result.data` — array of rows: `[timestamp, [col1_values], [col2_values], ...]`

524 +

525 +Each value array contains 3 elements:

526 +- **Index 0 (`value`)**: The metric value

527 +- **Index 1 (`arp`)**: Anomaly rate (0-100). Percentage of raw samples in this interval flagged as anomalous by ML

528 +- **Index 2 (`pa`)**: Partial data. Non-zero means the interval has incomplete data (e.g., at query boundaries)

529 +

530 +### db

531 +

532 +| Field | Description |

533 +|-------|-------------|

534 +| `tiers` | Number of database tiers |

535 +| `update_every` | Maximum update interval across nodes |

536 +| `first_entry` | Earliest data timestamp |

537 +| `last_entry` | Latest data timestamp |

538 +| `per_tier[]` | Per-tier info: `tier`, `queries`, `points`, `update_every`, `first_entry`, `last_entry` |

539 +| `units` | Database units |

540 +| `dimensions.ids` | Database dimension IDs |

541 +| `dimensions.units` | Database dimension units |

542 +| `dimensions.sts` | Database-level stats |

543 +

544 +### timings

545 +

546 +| Field | Description |

547 +|-------|-------------|

548 +| `total_ms` | Total query time |

549 +| `routing_ms` | Time to route to agents |

550 +| `prep_ms` | Preparation time (per agent) |

551 +| `query_ms` | Query execution time (per agent) |

552 +| `output_ms` | Output formatting time (per agent) |

553 +| `node_max_ms` | Slowest node response time |

554 +| `cloud_ms` | Cloud processing time |

555 +

556 +---

557 +

558 +## How Users Find Metric Names in the UI

559 +

560 +1. **Context names** (for `scope.contexts`):

561 + - The context is shown next to the chart title (e.g., `system.cpu`, `disk.space`). You can click it to copy it.

562 + - Use the **Metrics** tab in the dashboard to browse all available contexts

563 + - Use the `/contexts` endpoint with `scope.contexts: ["pattern*"]`

564 +

565 +2. **Dimension names** (for `scope.dimensions`):

566 + - Visible in the chart legend (e.g., `user`, `system`, `iowait` for CPU)

567 + - Query with `group_by: ["dimension"]` to see all dimension names in `view.dimensions.names`

568 +

569 +3. **Node hostnames and UUIDs** (for `scope.nodes`):

570 + - The **Nodes** tab lists hostnames

571 + - Use `/nodes` endpoint to get UUIDs (the `nd` field)

572 +

573 +4. **Labels** (for `scope.labels` and `group_by_label`):

574 + - Click the labels drop-down on a chart, to see all label keys and values

575 + - Labels like `mount_point`, `filesystem`, `interface` appear in the list

576 + - Query with `group_by: ["selected"]` and check `summary.labels` in the response to discover available label keys and values for a context

577 +

578 +---

579 +

580 +## Practical Examples

581 +

582 +All examples use this pattern — users replace the 3 variables at the top:

583 +

584 +```bash

585 +TOKEN="YOUR_API_TOKEN"

586 +SPACE="YOUR_SPACE_ID"

587 +ROOM="YOUR_ROOM_ID"

588 +```

589 +

590 +### Example 1: Total CPU Across All Nodes (Last 10 Minutes)

591 +

592 +```bash

593 +TOKEN="YOUR_API_TOKEN"

594 +SPACE="YOUR_SPACE_ID"

595 +ROOM="YOUR_ROOM_ID"

596 +

597 +read -r -d '' PAYLOAD <<'EOF'

598 +{

599 + "scope": {"contexts": ["system.cpu"]},

600 + "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},

601 + "window": {"after": -600, "before": 0, "points": 5},

602 + "aggregations": {

603 + "metrics": [{"group_by": ["selected"], "aggregation": "sum"}],

604 + "time": {"time_group": "average"}

605 + },

606 + "format": "json2",

607 + "options": ["jsonwrap", "minify"],

608 + "timeout": 30000

609 +}

610 +EOF

611 +

612 +curl -s -X POST \

613 + -H 'Content-Type: application/json' \

614 + -H "Authorization: Bearer $TOKEN" \

615 + "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \

616 + -d "$PAYLOAD"

617 +```

618 +

619 +Result: Single column `selected` with total CPU % (sum of all dimensions across all nodes) at 5 time points.

620 +

621 +### Example 2: CPU Breakdown by Dimension

622 +

623 +```bash

624 +TOKEN="YOUR_API_TOKEN"

625 +SPACE="YOUR_SPACE_ID"

626 +ROOM="YOUR_ROOM_ID"

627 +

628 +read -r -d '' PAYLOAD <<'EOF'

629 +{

630 + "scope": {"contexts": ["system.cpu"]},

631 + "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},

632 + "window": {"after": -600, "before": 0, "points": 5},

633 + "aggregations": {

634 + "metrics": [{"group_by": ["dimension"], "aggregation": "sum"}],

635 + "time": {"time_group": "average"}

636 + },

637 + "format": "json2",

638 + "options": ["jsonwrap", "minify"],

639 + "timeout": 30000

640 +}

641 +EOF

642 +

643 +curl -s -X POST \

644 + -H 'Content-Type: application/json' \

645 + -H "Authorization: Bearer $TOKEN" \

646 + "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \

647 + -d "$PAYLOAD"

648 +```

649 +

650 +Result: One column per dimension (`user`, `system`, `iowait`, `irq`, `softirq`, `steal`, `guest`, `nice`). Values are summed across all nodes.

651 +

652 +### Example 3: Compare CPU Per Node

653 +

654 +```bash

655 +TOKEN="YOUR_API_TOKEN"

656 +SPACE="YOUR_SPACE_ID"

657 +ROOM="YOUR_ROOM_ID"

658 +

659 +read -r -d '' PAYLOAD <<'EOF'

660 +{

661 + "scope": {"contexts": ["system.cpu"]},

662 + "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},

663 + "window": {"after": -600, "before": 0, "points": 5},

664 + "aggregations": {

665 + "metrics": [{"group_by": ["node"], "aggregation": "sum"}],

666 + "time": {"time_group": "average"}

667 + },

668 + "format": "json2",

669 + "options": ["jsonwrap", "minify"],

670 + "timeout": 30000

671 +}

672 +EOF

673 +

674 +curl -s -X POST \

675 + -H 'Content-Type: application/json' \

676 + -H "Authorization: Bearer $TOKEN" \

677 + "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \

678 + -d "$PAYLOAD"

679 +```

680 +

681 +Result: One column per node hostname. Values are total CPU % per node. Column names in `view.dimensions.names`.

682 +

683 +### Example 4: Peak CPU Per Node Over Last Hour

684 +

685 +```bash

686 +TOKEN="YOUR_API_TOKEN"

687 +SPACE="YOUR_SPACE_ID"

688 +ROOM="YOUR_ROOM_ID"

689 +

690 +read -r -d '' PAYLOAD <<'EOF'

691 +{

692 + "scope": {"contexts": ["system.cpu"]},

693 + "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},

694 + "window": {"after": -3600, "before": 0, "points": 6},

695 + "aggregations": {

696 + "metrics": [{"group_by": ["node"], "aggregation": "max"}],

697 + "time": {"time_group": "max"}

698 + },

699 + "format": "json2",

700 + "options": ["jsonwrap", "minify"],

701 + "timeout": 30000

702 +}

703 +EOF

704 +

705 +curl -s -X POST \

706 + -H 'Content-Type: application/json' \

707 + -H "Authorization: Bearer $TOKEN" \

708 + "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \

709 + -d "$PAYLOAD"

710 +```

711 +

712 +Result: 6 points (10-min intervals). Each value is the **peak** CPU for that node in that interval.

713 +

714 +### Example 5: Disk Space Grouped by Filesystem Type

715 +

716 +```bash

717 +TOKEN="YOUR_API_TOKEN"

718 +SPACE="YOUR_SPACE_ID"

719 +ROOM="YOUR_ROOM_ID"

720 +

721 +read -r -d '' PAYLOAD <<'EOF'

722 +{

723 + "scope": {"contexts": ["disk.space"]},

724 + "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},

725 + "window": {"after": -600, "before": 0, "points": 5},

726 + "aggregations": {

727 + "metrics": [{"group_by": ["label"], "group_by_label": ["filesystem"], "aggregation": "sum"}],

728 + "time": {"time_group": "average"}

729 + },

730 + "format": "json2",

731 + "options": ["jsonwrap", "minify"],

732 + "timeout": 30000

733 +}

734 +EOF

735 +

736 +curl -s -X POST \

737 + -H 'Content-Type: application/json' \

738 + -H "Authorization: Bearer $TOKEN" \

739 + "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \

740 + -d "$PAYLOAD"

741 +```

742 +

743 +Result: One column per filesystem type (ext4, btrfs, tmpfs, etc.). Values are total disk space summed across all nodes.

744 +

745 +### Example 6: Filter by Specific Nodes (UUIDs)

746 +

747 +```bash

748 +TOKEN="YOUR_API_TOKEN"

749 +SPACE="YOUR_SPACE_ID"

750 +ROOM="YOUR_ROOM_ID"

751 +

752 +read -r -d '' PAYLOAD <<'EOF'

753 +{

754 + "scope": {"contexts": ["system.cpu"], "nodes": ["NODE_UUID_1", "NODE_UUID_2"]},

755 + "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},

756 + "window": {"after": -600, "before": 0, "points": 5},

757 + "aggregations": {

758 + "metrics": [{"group_by": ["node"], "aggregation": "sum"}],

759 + "time": {"time_group": "average"}

760 + },

761 + "format": "json2",

762 + "options": ["jsonwrap", "minify"],

763 + "timeout": 30000

764 +}

765 +EOF

766 +

767 +curl -s -X POST \

768 + -H 'Content-Type: application/json' \

769 + -H "Authorization: Bearer $TOKEN" \

770 + "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \

771 + -d "$PAYLOAD"

772 +```

773 +

774 +Result: Data and metadata scoped to only those 2 nodes. First call `/nodes` to get UUIDs (the `nd` field).

775 +

776 +### Example 7: Filter by Labels

777 +

778 +```bash

779 +TOKEN="YOUR_API_TOKEN"

780 +SPACE="YOUR_SPACE_ID"

781 +ROOM="YOUR_ROOM_ID"

782 +

783 +read -r -d '' PAYLOAD <<'EOF'

784 +{

785 + "scope": {"contexts": ["disk.space"], "labels": ["mount_point:/", "filesystem:ext4"]},

786 + "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},

787 + "window": {"after": -600, "before": 0, "points": 5},

788 + "aggregations": {

789 + "metrics": [{"group_by": ["selected"], "aggregation": "sum"}],

790 + "time": {"time_group": "average"}

791 + },

792 + "format": "json2",

793 + "options": ["jsonwrap", "minify"],

794 + "timeout": 30000

795 +}

796 +EOF

797 +

798 +curl -s -X POST \

799 + -H 'Content-Type: application/json' \

800 + -H "Authorization: Bearer $TOKEN" \

801 + "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \

802 + -d "$PAYLOAD"

803 +```

804 +

805 +Result: Only ext4 root mount points. Multiple labels with different keys are AND-combined.

806 +

807 +### Example 8: Filter Specific Dimensions

808 +

809 +```bash

810 +TOKEN="YOUR_API_TOKEN"

811 +SPACE="YOUR_SPACE_ID"

812 +ROOM="YOUR_ROOM_ID"

813 +

814 +read -r -d '' PAYLOAD <<'EOF'

815 +{

816 + "scope": {"contexts": ["system.cpu"], "dimensions": ["user", "system"]},

817 + "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},

818 + "window": {"after": -600, "before": 0, "points": 5},

819 + "aggregations": {

820 + "metrics": [{"group_by": ["dimension"], "aggregation": "sum"}],

821 + "time": {"time_group": "average"}

822 + },

823 + "format": "json2",

824 + "options": ["jsonwrap", "minify"],

825 + "timeout": 30000

826 +}

827 +EOF

828 +

829 +curl -s -X POST \

830 + -H 'Content-Type: application/json' \

831 + -H "Authorization: Bearer $TOKEN" \

832 + "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \

833 + -d "$PAYLOAD"

834 +```

835 +

836 +Result: Only `user` and `system` CPU dimensions.

837 +

838 +### Example 9: Discover Labels for a Context

839 +

840 +```bash

841 +TOKEN="YOUR_API_TOKEN"

842 +SPACE="YOUR_SPACE_ID"

843 +ROOM="YOUR_ROOM_ID"

844 +

845 +read -r -d '' PAYLOAD <<'EOF'

846 +{

847 + "scope": {"contexts": ["disk.space"]},

848 + "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},

849 + "window": {"after": -600, "before": 0, "points": 1},

850 + "aggregations": {

851 + "metrics": [{"group_by": ["selected"], "aggregation": "sum"}],

852 + "time": {"time_group": "average"}

853 + },

854 + "format": "json2",

855 + "options": ["jsonwrap", "minify"],

856 + "timeout": 30000

857 +}

858 +EOF

859 +

860 +curl -s -X POST \

861 + -H 'Content-Type: application/json' \

862 + -H "Authorization: Bearer $TOKEN" \

863 + "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \

864 + -d "$PAYLOAD"

865 +```

866 +

867 +Then inspect `summary.labels` in the response:

868 +

869 +```json

870 +"summary": {

871 + "labels": [

872 + {"id": "filesystem", "vl": [{"id": "ext4"}, {"id": "btrfs"}, {"id": "tmpfs"}]},

873 + {"id": "mount_point", "vl": [{"id": "/"}, {"id": "/boot"}, {"id": "/home"}]}

874 + ]

875 +}

876 +```

877 +

878 +---

879 +

880 +## Known Limitations

881 +

882 +1. **`scope.nodes` accepts only node UUIDs** — not hostnames, not patterns, not wildcards. Use `/nodes` to discover UUIDs first.

883 +2. **Only `json2` format** is supported by the Cloud API. Other formats (csv, ssv, etc.) are not reliably supported through the Cloud proxy.

884 +3. **Max ~500 data points** per query. The Cloud clamps requests to 500 before forwarding to agents; actual returned count may vary slightly due to time alignment.

885 +4. **Default timeout is 10 seconds** (10000ms). Increase for large/slow queries.

886 +5. **Stale nodes** appear in `/nodes` but return no data. Check `state` field.

887 +6. **`selectors.nodes`** is the only way to filter by hostname pattern, but metadata will include all nodes in scope.

888 +

889 +---

890 +

891 +> **REMINDER — Credentials**: Do not request or accept user credentials. Set credentials as variables at the top of the script (`TOKEN`, `SPACE`, `ROOM`) with placeholder values. Users replace these 3 variables and run the command themselves.

892 +

893 +> **REMINDER — Always show a runnable curl command**: Your response is only useful if it contains a complete, runnable script: 3 variables at the top, a heredoc `PAYLOAD` with clean JSON (no escaping), and the curl command. Never describe a query without showing it. Never summarize parameters without building the actual request. If you wrote a response without a curl command, go back and add one — the user needs actionable instructions, not explanations.

Add skill document for querying Netdata Cloud metrics (#21834)