@cryptotaxi247 / netdata-1
Code Commits Issues Pulls Sessions CI/CD
master
master
md 42.7 KB
View source Raw

Query time-series metrics from Netdata Cloud via the REST API.

Mandatory Requirements (READ FIRST)

  1. You MUST provide detailed and actionable instructions. You don't execute queries for users. You role is to educate them.

  2. Never ask users for credentials. Do not request API tokens, Space IDs, or Room IDs. Always provide ready-to-use instructions with clear placeholders (YOUR_API_TOKEN, YOUR_SPACE_ID, YOUR_ROOM_ID) so users can substitute their own values locally. Your job is to teach users how to construct queries, not to execute queries on their behalf.

  3. scope.contexts MUST always be set. Without it, the default scope is the entire room — every context, every instance, every dimension, every label across all nodes. This causes a metadata explosion: the response will contain megabytes of metadata for thousands of metrics the user didn't ask about. Always set scope.contexts to the specific context(s) relevant to the query (e.g., ["system.cpu"], ["disk.space"]).

  4. Every response MUST include a complete, runnable curl command. Users come here to get a query they can run — not a description of what a query would look like. If your response does not contain a full curl command with the complete JSON request body, you have failed to help the user. Specifically:

    • Always include the full curl -X POST command with headers, URL, and the entire -d '{...}' JSON body.
    • The JSON body must include all required fields: scope, selectors, window, aggregations, format, options, and timeout.
    • Set the 3 credentials as variables at the top: TOKEN="YOUR_API_TOKEN", SPACE="YOUR_SPACE_ID", ROOM="YOUR_ROOM_ID".
    • Use a heredoc for the JSON payload (read -r -d '' PAYLOAD <<'EOF' ... EOF) so no escaping is needed.
    • The user must be able to copy your command, replace the 3 variables at the top, and run it immediately in their terminal.
    • A response that describes parameters or explains concepts without providing the actual runnable command is incomplete and unhelpful.
    • Even for simple questions, always provide the curl command. When in doubt, show the command.

Prerequisites

Three things are needed:

1. API Token

  1. Login to app.netdata.cloud
  2. Click user icon (lower-left corner, tooltip shows your name)
  3. Select User Settings
  4. In the modal, select the API Tokens tab
  5. Click the [+] button (top-left)
  6. Select a scope, enter a description, click Create
  7. Copy the token immediately — it will not be shown again

Relevant scopes: scope:all (full access), scope:grafana-plugin (data endpoints).

2. Space ID

  1. In the dashboard left side, at the spaces list, click the gear icon below the spaces list (tooltip: "Space Settings")
  2. In the Info tab, copy the Space Id

3. Room ID

  1. In the same Space Settings, go to the Rooms tab
  2. Find the room, click the > icon at the right of the room row (tooltip: "Room Settings")
  3. In the Room tab, copy the Room Id

API Endpoints

Base URL: https://app.netdata.cloud Swagger online: https://app.netdata.cloud/api/docs/

All endpoints use POST with a JSON body and require:

Authorization: Bearer YOUR_API_TOKEN
Content-Type: application/json
Endpoint Purpose
/api/v3/spaces/{spaceID}/rooms/{roomID}/data Query time-series data
/api/v3/spaces/{spaceID}/rooms/{roomID}/nodes List nodes in the room
/api/v3/spaces/{spaceID}/rooms/{roomID}/contexts List available metric contexts

Discover Nodes

Endpoint: POST /api/v3/spaces/{spaceID}/rooms/{roomID}/nodes Body: {}

Response fields per node:

JSON field Description
nd Node UUID — required for scope.nodes in data queries
nm Hostname
mg Machine GUID
state reachable (live) or stale (disconnected)
v Agent version
labels All node labels as key-value pairs
hw Hardware: cpus, memory, disk_space, architecture
os OS: nm (name), v (version), kernel
health Alert summary: status, alerts.warning, alerts.critical
capabilities Supported features: ml, funcs, health, etc.

Example:

TOKEN="YOUR_API_TOKEN"
SPACE="YOUR_SPACE_ID"
ROOM="YOUR_ROOM_ID"

curl -s -X POST \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/nodes" \
  -d '{}'

Discover Contexts

Contexts are metric types (e.g., system.cpu, disk.space, net.net).

Endpoint: POST /api/v3/spaces/{spaceID}/rooms/{roomID}/contexts

{
  "scope": { "contexts": ["system.*"] },
  "selectors": { "nodes": ["*"], "contexts": ["*"] }
}

scope.contexts supports patterns: system.*, disk.*, *cpu*.

Query Metric Data

Endpoint: POST /api/v3/spaces/{spaceID}/rooms/{roomID}/data

Full Request Body Structure

{
  "scope": {
    "nodes": [],
    "contexts": ["REQUIRED — e.g. system.cpu, disk.space"],
    "instances": [],
    "dimensions": [],
    "labels": []
  },
  "selectors": {
    "nodes": ["*"],
    "contexts": ["*"],
    "instances": ["*"],
    "dimensions": ["*"],
    "labels": ["*"],
    "alerts": ["*"]
  },
  "window": {
    "after": 0,
    "before": 0,
    "points": 0,
    "duration": 0,
    "tier": null,
    "baseline": null
  },
  "aggregations": {
    "metrics": [
      {
        "group_by": [],
        "group_by_label": [],
        "aggregation": "avg"
      }
    ],
    "time": {
      "time_group": "average",
      "time_group_options": null,
      "time_resampling": null
    }
  },
  "format": "json2",
  "options": ["jsonwrap", "minify", "unaligned"],
  "timeout": 10000,
  "limit": null
}

scope — Define the Data Universe

Scope controls both data and metadata in the response. Use scope fields for filtering so that the response metadata is focused on what you asked for.

WARNING: The default scope (when fields are omitted) is all nodes and all contexts in the room. This can produce multi-megabyte responses with metadata for thousands of metrics. scope.contexts MUST always be set to avoid this metadata explosion.

Field Type Accepts Default (if omitted)
nodes string[] Node UUIDs only (the nd field from /nodes) All nodes in the room
contexts string[] Exact names or patterns (system.*, *cpu*) REQUIRED — always set to avoid metadata explosion
instances string[] Exact names or patterns (disk_space./@NODE_UUID) All instances
dimensions string[] Exact names or patterns (*user*, sent) All dimensions
labels string[] key:value pairs (filesystem:btrfs, mount_point:/) No label filter

Multiple entries in the same field are OR-combined. Multiple labels entries with different keys are AND-combined.

Filtering by node: Use selectors.nodes with hostname patterns (e.g., ["web*", "prod-*"]). This is the simplest and preferred approach. Metadata will include all nodes in the room, but data will be filtered correctly.

Advanced: scope.nodes restricts both data AND metadata, but it only accepts node UUIDs (the nd field from /nodes). Hostnames, patterns, and wildcards do not work. Use this only when you need tight metadata scoping — otherwise prefer selectors.nodes.

CRITICAL: scope.contexts MUST always be set. The context is the metric type shown next to the chart title on the Netdata dashboard (e.g., system.cpu, disk.space). Clicking it copies it to the clipboard.

selectors — Further Filter Data Within the Scope

Selectors filter data only — response metadata still reflects the full scope (the room). For programmatic API queries, use scope for filtering and set all selectors to ["*"].

Selectors exist for the Netdata dashboard, which needs full metadata to show context ("the whole") while displaying a filtered subset.

Field Type Checked against Supports
nodes string[] Machine GUID, node ID, hostname Simple patterns, positive and negative
contexts string[] Context ID Simple patterns, positive and negative
instances string[] Instance ID, instance name, instance@machine_guid Simple patterns, positive and negative
dimensions string[] Dimension ID and dimension name Simple patterns, positive and negative
labels string[] name:value of all labels Simple patterns (negative not recommended)
alerts string[] Alert name, name:status (CLEAR, WARNING, CRITICAL, REMOVED, UNDEFINED, UNINITIALIZED) Simple patterns; negative excludes instances

selectors.nodes is the preferred way to filter by node. It accepts hostname patterns (e.g., ["web*", "!staging*"]), making it simpler than looking up UUIDs for scope.nodes. Metadata will include all nodes in scope, but data is filtered correctly.

CRITICAL: scope.contexts MUST always be set to avoid metadata explosion.

window — Time Range

Field Type Description Default
after int Start time. Negative = relative seconds from before (max -94608000 = 3 years). Positive = Unix epoch. -600
before int End time. Negative = relative seconds from now (max -94608000). Positive = Unix epoch. 0 (now)
points int Number of data points to return. 0 or omitted = all available points. 0
duration int Alternative to after/before. Duration in seconds. 0
tier int? Force a specific dbengine storage tier (0 = per-second, 1 = per-minute, 2 = per-hour). null = auto-select. null
baseline object? Baseline window for comparison queries. Same fields as window: after, before, points, duration. null

Max points requested: approximately 500 (ScopeDataRequestMaxPoints). The Cloud clamps the request to 500 before forwarding to agents, but the actual number returned may vary slightly due to time alignment.

The time range is divided into points equal intervals. Each interval is aggregated using the time_group function.

Time resolution: duration ÷ points = seconds per point

This is the most common assistant mistake. The number of points does NOT mean "give me per-second data". It means "split the duration into N equal buckets". The actual time resolution per point is:

seconds_per_point = abs(duration) ÷ points

To get per-second data, set points equal to the duration in seconds.

Examples:

You want Set after Set points Result
Per-second resolution, last 2 minutes -120 120 1 second per point
Per-second resolution, last 5 minutes -300 300 1 second per point
10-second buckets, last 10 minutes -600 60 10 seconds per point
Per-minute resolution, last hour -3600 60 60 seconds per point

Common mistake: requesting after: -600, points: 30 and expecting per-second data. Result: 600 ÷ 30 = 20 seconds per point (heavily aggregated). Per-second data over 10 minutes requires after: -600, points: 600 (which is at the 500-point cap; either request 8 minutes 20 seconds at 500 points, or accept a slightly coarser resolution).

Per-second data also requires that the dbengine tier 0 (per-second storage) covers the requested time range. If the agent's tier 0 retention is shorter than abs(after), the engine auto-selects a coarser tier (per-minute or per-hour). Force tier 0 with "tier": 0 in the window if you need to assert per-second data is actually available -- the query will fail rather than silently downsample.

points: 0 (the default) is NOT "per-second" -- it requests "all available points", which is whatever the engine returns within its 500-point cap and the storage tier's natural granularity. For a 1-hour query against tier-0 storage, the engine still aggregates because 3600 > 500.

How the Query Pipeline Works

The query engine is a pipeline with two aggregation stages:

  1. Identify time-series matching the scope and selectors
  2. Set up the output time-series based on group_by (e.g., 2 groups for label values A and B)
  3. For each matched time-series:
    • Stage 1 — Time aggregation (time_group): Aggregate raw samples within each time interval into points data points (e.g., average 86400 per-second samples into 100 points)
    • Stage 2 — Metric aggregation (aggregation): Add the time-aggregated points into the appropriate output time-series using the aggregation function (e.g., SUM into group A or B)
  4. Present the grouped, aggregated result

Key insight: time_group reduces samples within each time-series. aggregation combines multiple time-series into groups. They operate in sequence — metric aggregation works on already time-aggregated data.

Example: 1000 containers, group by label namespace (2 values: A and B), 100 points over 1 day

  1. Output setup: 2 time-series needed (A and B), each with 100 points
  2. For each of the 1000 container time-series:
    • Time-aggregate 86400 seconds into 100 points using time_group (e.g., average)
    • Add those 100 points into either A or B using aggregation (e.g., sum)
  3. Result: 2 columns (A, B) × 100 rows

Choosing time_group Based on What the User Wants

User intent time_group Why
Average resource consumption (rate metrics: CPU, I/O, bandwidth) average Rate metrics represent per-second rates; averaging preserves the rate
Average resource consumption (gauge metrics: memory, disk space, connections) average or max Gauges represent current state; max shows peak usage
Find spikes or peaks (any metric type) max Captures the highest value within each interval
Total volume transferred (counters: bytes, packets) sum Sums the actual volume
Count events matching a condition countif Counts samples matching a threshold

Choosing aggregation Based on How to Combine Series

User intent aggregation Why
Total across all series (e.g., total CPU across all containers) sum Adds up all contributions
Average across series avg Mean of the group
Worst case across series max Highest value in the group
Best case across series min Lowest value in the group

Mapping User Questions to Parameters

"Find a CPU spike over the last week across all my containers" → time_group: "max", aggregation: "sum" (sum user+system), group_by: ["instance"]

"Which namespace consumed most CPU over the last week?" → time_group: "average" (per-second rate) or "sum" (total), aggregation: "sum", group_by: ["label"], group_by_label: ["namespace"]

"Peak memory usage per node over the last 24 hours" → time_group: "max" (gauge metric, want peak), aggregation: "sum", group_by: ["node"]

Research the Context Before Answering

Before constructing a query for a user, you should understand the metric context they are asking about — its dimensions, labels, and whether it represents rates (incremental) or gauges (absolute). Search for the context name (e.g., cgroup.cpu, disk.space, nginx.connections) in the Netdata source code to find its metadata.yaml, which defines dimensions, units, chart type, and available labels. This ensures you choose the correct time_group and aggregation for their use case.

aggregations.time — Time Aggregation

Controls how raw data points within each time interval are combined into one value per series.

Field Type Description Default
time_group string Aggregation function (see table below) average
time_group_options string? Additional parameter for the function null
time_resampling int? Resample "per-second" values to "per-minute" (60) or "per-hour" (3600). Only works with time_group=average. null

time_group values

Value Aliases Description
average avg Mean value (default)
min Minimum value
max Maximum value
sum Sum of values
median Median value
stddev Standard deviation
cv Coefficient of variation (stddev/mean)
ses Single exponential smoothing
des Double exponential smoothing
incremental-sum Difference between last and first value in interval
countif Count values matching condition. Set condition in time_group_options: ">0", "=0", "!=0", "<=10"
percentile Percentile. Set percentile value in time_group_options: "95", "99"
trimmed-mean Mean after trimming outliers. Set trim % in time_group_options
trimmed-median Median after trimming outliers. Set trim % in time_group_options

:::important

When using time_group values other than min, max, average, or sum, you MUST specify "tier": 0 in the window object to ensure a non-aggregated storage tier is used. Without it, the query may use a pre-aggregated tier (per-minute or per-hour) where advanced functions like median, stddev, ses, des, percentile, countif, trimmed-mean, trimmed-median, and extremes cannot work correctly.

:::

time_group_options values

Used with Value format Example
countif Comparison operator + value ">0", "=0", "!=0", "<=100"
percentile Percentile value (0-100) "95", "99.5"
trimmed-mean Trim percentage "5", "10"
trimmed-median Trim percentage "5", "10"

:::important

When using time_group values other than min, max, average, or sum, you MUST specify "tier": 0 in the window object to ensure a non-aggregated storage tier is used. Without it, the query may use a pre-aggregated tier (per-minute or per-hour) where advanced functions like median, stddev, ses, des, percentile, countif, trimmed-mean, trimmed-median, and extremes cannot work correctly.

:::

aggregations.metrics[] — Dimension Aggregation

Controls how multiple time-series are combined. Each entry defines a grouping pass. At least one is required.

Field Type Description Default
group_by string[] What to group by (see table below) (required)
group_by_label string[] Label keys to group by. Required when group_by includes label. Order is respected. []
aggregation string How to combine grouped values (see table below) average

group_by values

All values can be combined together except selected (if selected is present, all others are ignored).

Value Result columns represent Use case
selected Single column: all matched data combined into one series Total/aggregate value across everything
dimension One column per unique dimension name Break down by metric component (user/system/iowait for CPU)
node One column per node Compare nodes side by side
instance One column per instance (context@hostname) Compare instances across nodes
label One column per unique label value Group by label (requires group_by_label)
context One column per context Compare different metric types
units One column per unit type Group by measurement unit
percentage-of-instance Percentages per dimension within each instance Show proportions instead of absolutes

Combination example: "group_by": ["node", "dimension"] creates one column per node+dimension combination.

aggregation values

Value Aliases Description
avg average Mean of grouped values (default)
sum Sum of grouped values
min Minimum of grouped values
max Maximum of grouped values
median Median of grouped values
percentage Express as percentage of total

format

Only json2 is supported by Netdata Cloud.

options

Array of strings. Each option modifies the response behavior.

Option Description
jsonwrap Recommended. Wraps the result with metadata (summary, view, db, timings)
minify Recommended. Minimizes JSON output size
unaligned Recommended for API queries. Without this, time intervals are aligned to wall-clock boundaries based on the requested period (e.g., 1-hour queries snap to 00:00–01:00). This is useful for dashboards (prevents charts from "dancing" on refresh) but confusing for API users who expect data for the exact time range they requested. Always use unaligned for programmatic queries.
nonzero Exclude dimensions that have only zero values
null2zero Replace null values with zero
abs Return the absolute value of all data
absolute Same as abs
display-absolute Display absolute values
flip Flip the sign of values (multiply by -1)
reversed Reverse the order of data points (oldest last)
min2max Show the range (max - min) instead of the value
percentage Convert values to percentages
seconds Return timestamps as seconds
ms Return timestamps as milliseconds
milliseconds Same as ms
match-ids Match dimensions by ID only (not name)
match-names Match dimensions by name only (not ID)
anomaly-bit Return anomaly rate instead of metric values
natural-points Return natural data points (one per collection interval)
virtual-points Return virtual (interpolated) data points
objectrows Return data rows as objects instead of arrays
google_json Format compatible with Google Charts

Recommended minimum: ["jsonwrap", "minify", "unaligned"]

timeout

Query timeout in milliseconds. Default: 10000 (10 seconds). Set higher for queries spanning many nodes or long time ranges.

limit

Optional integer. Limits the number of dimensions returned. Cannot be negative. Useful when querying high-cardinality contexts.

Response Structure

With jsonwrap option, the response contains:

Top-level fields

Field Description
api API version (integer)
agents List of agents consulted
versions Hash values to detect database changes
summary Metadata about nodes, contexts, instances, dimensions, labels, alerts
totals Counts of selected/excluded/queried items
functions List of supported functions
db Database info (tiers, retention, update frequency)
view Presentation metadata (title, units, dimensions, time range)
result The actual time-series data
timings Query performance metrics

summary

Metadata determined by scope. Statistics within are influenced by selectors.

summary.nodes[]     — ni (index), mg (machine GUID), nd (node UUID), nm (hostname), st (status), sts (stats)
summary.contexts[]  — id, is (instances count), ds (dimensions count), al (alerts), sts (stats)
summary.instances[] — id, nm (name), ni (node index), ds (dimensions count), al (alerts), sts (stats)
summary.dimensions[] — id, nm (name), ds (count), pri (priority), sts (stats)
summary.labels[]    — id (label key), vl[] (label values with id and stats)
summary.alerts[]    — nm (name), cl (clear count), wr (warning count), cr (critical count)

Stats object (sts): min, max, avg (average), arp (anomaly rate %), con (contribution %).

ItemsCount fields: sl (selected), ex (excluded), qr (query success), fl (query fail).

view

Field Description
title Chart title
update_every Data collection interval (seconds)
after Actual start timestamp of returned data
before Actual end timestamp of returned data
points Number of data points returned
units Unit of measurement
chart_type Default chart type (line, area, stacked)
min Minimum value across all data
max Maximum value across all data
dimensions.grouped_by Array confirming the group_by used
dimensions.ids Unique dimension IDs
dimensions.names Human-readable dimension names (column headers)
dimensions.units Units per dimension
dimensions.priorities Display priority per dimension
dimensions.aggregated Number of source metrics aggregated into each dimension
dimensions.sts Stats arrays per dimension: min[], max[], avg[], arp[], con[]

result — The Time-Series Data

{
  "labels": ["time", "host1", "host2"],
  "point": {"value": 0, "arp": 1, "pa": 2},
  "data": [
    [1700000060, [5.23, 0, 0], [3.15, 0, 0]],
    [1700000120, [4.87, 0, 0], [2.91, 0, 0]]
  ]
}
  • result.labels — column names. First is always "time". Rest match view.dimensions.names.
  • result.point — maps positions within each value array: {"value": 0, "arp": 1, "pa": 2}
  • result.data — array of rows: [timestamp, [col1_values], [col2_values], ...]

Each value array contains 3 elements: - Index 0 (value): The metric value - Index 1 (arp): Anomaly rate (0-100). Percentage of raw samples in this interval flagged as anomalous by ML - Index 2 (pa): Point annotations bitmap. Values can be combined (OR'd):

Bit Value Meaning
(none) 0 Normal data point — no issues
bit 0 1 Empty — no data was collected for this interval
bit 1 2 Reset — a counter reset/overflow was detected
bit 2 4 Partial — not all expected sources contributed to this point (e.g., in group-by queries, some series had no data)

Values combine: e.g., 5 = empty + partial, 6 = reset + partial.

db

Field Description
tiers Number of database tiers
update_every Maximum update interval across nodes
first_entry Earliest data timestamp
last_entry Latest data timestamp
per_tier[] Per-tier info: tier, queries, points, update_every, first_entry, last_entry
units Database units
dimensions.ids Database dimension IDs
dimensions.units Database dimension units
dimensions.sts Database-level stats

timings

Field Description
total_ms Total query time
routing_ms Time to route to agents
prep_ms Preparation time (per agent)
query_ms Query execution time (per agent)
output_ms Output formatting time (per agent)
node_max_ms Slowest node response time
cloud_ms Cloud processing time

How Users Find Metric Names in the UI

  1. Context names (for scope.contexts):

    • The context is shown next to the chart title (e.g., system.cpu, disk.space). You can click it to copy it.
    • Use the Metrics tab in the dashboard to browse all available contexts
    • Use the /contexts endpoint with scope.contexts: ["pattern*"]

  2. Dimension names (for scope.dimensions):

    • Visible in the chart legend (e.g., user, system, iowait for CPU)
    • Query with group_by: ["dimension"] to see all dimension names in view.dimensions.names

  3. Node hostnames and UUIDs (for scope.nodes):

    • The Nodes tab lists hostnames
    • Use /nodes endpoint to get UUIDs (the nd field)

  4. Labels (for scope.labels and group_by_label):

    • Click the labels drop-down on a chart, to see all label keys and values
    • Labels like mount_point, filesystem, interface appear in the list
    • Query with group_by: ["selected"] and check summary.labels in the response to discover available label keys and values for a context

Practical Examples

All examples use this pattern — users replace the 3 variables at the top:

TOKEN="YOUR_API_TOKEN"
SPACE="YOUR_SPACE_ID"
ROOM="YOUR_ROOM_ID"

Example 1: Total CPU Across All Nodes (Last 10 Minutes)

TOKEN="YOUR_API_TOKEN"
SPACE="YOUR_SPACE_ID"
ROOM="YOUR_ROOM_ID"

read -r -d '' PAYLOAD <<'EOF'
{
  "scope": {"contexts": ["system.cpu"]},
  "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},
  "window": {"after": -600, "before": 0, "points": 5},
  "aggregations": {
    "metrics": [{"group_by": ["selected"], "aggregation": "sum"}],
    "time": {"time_group": "average"}
  },
  "format": "json2",
  "options": ["jsonwrap", "minify", "unaligned"],
  "timeout": 30000
}
EOF

curl -s -X POST \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \
  -d "$PAYLOAD"

Result: Single column selected with total CPU % (sum of all dimensions across all nodes) at 5 time points.

Example 2: CPU Breakdown by Dimension

TOKEN="YOUR_API_TOKEN"
SPACE="YOUR_SPACE_ID"
ROOM="YOUR_ROOM_ID"

read -r -d '' PAYLOAD <<'EOF'
{
  "scope": {"contexts": ["system.cpu"]},
  "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},
  "window": {"after": -600, "before": 0, "points": 5},
  "aggregations": {
    "metrics": [{"group_by": ["dimension"], "aggregation": "sum"}],
    "time": {"time_group": "average"}
  },
  "format": "json2",
  "options": ["jsonwrap", "minify", "unaligned"],
  "timeout": 30000
}
EOF

curl -s -X POST \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \
  -d "$PAYLOAD"

Result: One column per dimension (user, system, iowait, irq, softirq, steal, guest, nice). Values are summed across all nodes.

Example 3: Compare CPU Per Node

TOKEN="YOUR_API_TOKEN"
SPACE="YOUR_SPACE_ID"
ROOM="YOUR_ROOM_ID"

read -r -d '' PAYLOAD <<'EOF'
{
  "scope": {"contexts": ["system.cpu"]},
  "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},
  "window": {"after": -600, "before": 0, "points": 5},
  "aggregations": {
    "metrics": [{"group_by": ["node"], "aggregation": "sum"}],
    "time": {"time_group": "average"}
  },
  "format": "json2",
  "options": ["jsonwrap", "minify", "unaligned"],
  "timeout": 30000
}
EOF

curl -s -X POST \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \
  -d "$PAYLOAD"

Result: One column per node hostname. Values are total CPU % per node. Column names in view.dimensions.names.

Example 4: Peak CPU Per Node Over Last Hour

TOKEN="YOUR_API_TOKEN"
SPACE="YOUR_SPACE_ID"
ROOM="YOUR_ROOM_ID"

read -r -d '' PAYLOAD <<'EOF'
{
  "scope": {"contexts": ["system.cpu"]},
  "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},
  "window": {"after": -3600, "before": 0, "points": 6},
  "aggregations": {
    "metrics": [{"group_by": ["node"], "aggregation": "max"}],
    "time": {"time_group": "max"}
  },
  "format": "json2",
  "options": ["jsonwrap", "minify", "unaligned"],
  "timeout": 30000
}
EOF

curl -s -X POST \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \
  -d "$PAYLOAD"

Result: 6 points (10-min intervals). Each value is the peak CPU for that node in that interval.

Example 5: Disk Space Grouped by Filesystem Type

TOKEN="YOUR_API_TOKEN"
SPACE="YOUR_SPACE_ID"
ROOM="YOUR_ROOM_ID"

read -r -d '' PAYLOAD <<'EOF'
{
  "scope": {"contexts": ["disk.space"]},
  "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},
  "window": {"after": -600, "before": 0, "points": 5},
  "aggregations": {
    "metrics": [{"group_by": ["label"], "group_by_label": ["filesystem"], "aggregation": "sum"}],
    "time": {"time_group": "average"}
  },
  "format": "json2",
  "options": ["jsonwrap", "minify", "unaligned"],
  "timeout": 30000
}
EOF

curl -s -X POST \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \
  -d "$PAYLOAD"

Result: One column per filesystem type (ext4, btrfs, tmpfs, etc.). Values are total disk space summed across all nodes.

Example 6: Filter by Specific Nodes (UUIDs)

TOKEN="YOUR_API_TOKEN"
SPACE="YOUR_SPACE_ID"
ROOM="YOUR_ROOM_ID"

read -r -d '' PAYLOAD <<'EOF'
{
  "scope": {"contexts": ["system.cpu"], "nodes": ["NODE_UUID_1", "NODE_UUID_2"]},
  "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},
  "window": {"after": -600, "before": 0, "points": 5},
  "aggregations": {
    "metrics": [{"group_by": ["node"], "aggregation": "sum"}],
    "time": {"time_group": "average"}
  },
  "format": "json2",
  "options": ["jsonwrap", "minify", "unaligned"],
  "timeout": 30000
}
EOF

curl -s -X POST \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \
  -d "$PAYLOAD"

Result: Data and metadata scoped to only those 2 nodes. First call /nodes to get UUIDs (the nd field).

Example 7: Filter by Labels

TOKEN="YOUR_API_TOKEN"
SPACE="YOUR_SPACE_ID"
ROOM="YOUR_ROOM_ID"

read -r -d '' PAYLOAD <<'EOF'
{
  "scope": {"contexts": ["disk.space"], "labels": ["mount_point:/", "filesystem:ext4"]},
  "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},
  "window": {"after": -600, "before": 0, "points": 5},
  "aggregations": {
    "metrics": [{"group_by": ["selected"], "aggregation": "sum"}],
    "time": {"time_group": "average"}
  },
  "format": "json2",
  "options": ["jsonwrap", "minify", "unaligned"],
  "timeout": 30000
}
EOF

curl -s -X POST \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \
  -d "$PAYLOAD"

Result: Only ext4 root mount points. Multiple labels with different keys are AND-combined.

Example 8: Filter Specific Dimensions

TOKEN="YOUR_API_TOKEN"
SPACE="YOUR_SPACE_ID"
ROOM="YOUR_ROOM_ID"

read -r -d '' PAYLOAD <<'EOF'
{
  "scope": {"contexts": ["system.cpu"], "dimensions": ["user", "system"]},
  "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},
  "window": {"after": -600, "before": 0, "points": 5},
  "aggregations": {
    "metrics": [{"group_by": ["dimension"], "aggregation": "sum"}],
    "time": {"time_group": "average"}
  },
  "format": "json2",
  "options": ["jsonwrap", "minify", "unaligned"],
  "timeout": 30000
}
EOF

curl -s -X POST \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \
  -d "$PAYLOAD"

Result: Only user and system CPU dimensions.

Example 9: Discover Labels for a Context

TOKEN="YOUR_API_TOKEN"
SPACE="YOUR_SPACE_ID"
ROOM="YOUR_ROOM_ID"

read -r -d '' PAYLOAD <<'EOF'
{
  "scope": {"contexts": ["disk.space"]},
  "selectors": {"nodes": ["*"], "contexts": ["*"], "instances": ["*"], "dimensions": ["*"], "labels": ["*"], "alerts": ["*"]},
  "window": {"after": -600, "before": 0, "points": 1},
  "aggregations": {
    "metrics": [{"group_by": ["selected"], "aggregation": "sum"}],
    "time": {"time_group": "average"}
  },
  "format": "json2",
  "options": ["jsonwrap", "minify", "unaligned"],
  "timeout": 30000
}
EOF

curl -s -X POST \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  "https://app.netdata.cloud/api/v3/spaces/$SPACE/rooms/$ROOM/data" \
  -d "$PAYLOAD"

Then inspect summary.labels in the response:

"summary": {
  "labels": [
    {"id": "filesystem", "vl": [{"id": "ext4"}, {"id": "btrfs"}, {"id": "tmpfs"}]},
    {"id": "mount_point", "vl": [{"id": "/"}, {"id": "/boot"}, {"id": "/home"}]}
  ]
}

Known Limitations

  1. scope.contexts MUST always be set — without it, the response includes metadata for every metric in the room (hundreds of contexts, thousands of instances). This causes multi-megabyte responses.
  2. scope.nodes accepts only node UUIDs — use selectors.nodes with hostname patterns instead (simpler). Only use scope.nodes when you need tight metadata scoping.
  3. Only json2 format is supported by the Cloud API. Other formats (csv, ssv, etc.) are not reliably supported through the Cloud proxy.
  4. Max ~500 data points per query. The Cloud clamps requests to 500 before forwarding to agents; actual returned count may vary slightly due to time alignment.
  5. Default timeout is 10 seconds (10000ms). Increase for large/slow queries.
  6. Stale nodes appear in /nodes but return no data. Check state field.
  7. Always use unaligned option for API queries — without it, time intervals snap to wall-clock boundaries, which is confusing for programmatic use.

Troubleshooting / FAQ

Q: My query returns empty data — no error, but no results either. A: The API returns empty responses when nothing matches your filters. This is by design — it does not return an error. Check: (1) Is scope.contexts set to a valid context name? Typos like cpu.system instead of system.cpu silently return nothing. (2) Are your scope.labels or scope.dimensions correct? (3) If using scope.nodes, are the UUIDs valid? Use /contexts to verify context names and /nodes to verify UUIDs.

Q: The response is huge (megabytes) and slow. A: You are missing scope.contexts. Without it, the scope defaults to the entire room — every context, instance, dimension, and label across all nodes. Always set scope.contexts to only the context(s) you need (e.g., ["system.cpu"]).

Q: The time range in the response doesn't match what I requested. A: Add "unaligned" to the options array. Without it, time intervals are aligned to wall-clock boundaries based on the query period. For example, a 1-hour query might snap to 14:00–15:00 instead of 14:07–15:07. The unaligned option gives you the exact time range you asked for.

Q: I'm using hostnames in scope.nodes and getting no data. A: scope.nodes only accepts node UUIDs (the nd field from /nodes). Hostnames, patterns, and machine GUIDs do not work there. Use selectors.nodes instead — it accepts hostname patterns (e.g., ["web*", "prod-*"]) and is the preferred way to filter by node.

Q: I requested CSV format but got an error or garbled output. A: Only json2 format works through the Cloud API. The Cloud proxy cannot aggregate CSV responses from multiple agents. Always use "format": "json2".

Q: How do I find the context name for a metric I see on the dashboard? A: The context name is shown next to the chart title on every Netdata chart (e.g., system.cpu, disk.space, net.net). Click it to copy it to the clipboard. You can also use the /contexts endpoint with a pattern like ["system.*"] to browse available contexts.

Q: The anomaly rate (arp) is always 0 — is anomaly detection working? A: An arp of 0 means either no anomalies were detected (normal for healthy systems) or ML-based anomaly detection is disabled on the agent. Anomaly detection runs on every metric at collection time using ML (k-means clustering). Non-zero values indicate the percentage of raw samples in the interval that were flagged as anomalous. If arp is 0 across all metrics and all time ranges, the agent may have ML disabled.

Q: countif or percentile time_group returns unexpected values. A: These functions require raw per-second data. Add "tier": 0 to the window object to force the use of the non-aggregated storage tier. Without it, the query may use a pre-aggregated tier (per-minute or per-hour) where these functions cannot work correctly.

Q: I see non-zero pa values in the data — what do they mean? A: pa is a point annotations bitmap: 1 = empty (no data collected), 2 = counter reset/overflow detected, 4 = partial (not all sources contributed in a group-by query). Values combine: e.g., 5 = empty + partial. Non-zero pa values are common at query boundaries and during agent restarts.

Q: How do I filter nodes by hostname without looking up UUIDs? A: Use selectors.nodes with hostname patterns: "selectors": {"nodes": ["web*", "!staging*"], ...}. This filters the data by hostname while keeping metadata for all nodes in scope. It is simpler than looking up UUIDs for scope.nodes.

Q: Can I query multiple contexts in a single request? A: Yes. Set scope.contexts to multiple contexts, e.g., ["system.cpu", "system.ram"]. However, be careful with other filters — scope.dimensions, scope.labels, and selectors apply to all contexts in the query. A dimension filter like ["user"] would match the user dimension in system.cpu but might not exist in system.ram, causing that context to return no data. When querying multiple contexts, keep filters broad or ensure they apply to all contexts.

Q: My query timed out. A: The default timeout is 10 seconds (10000ms). For queries spanning many nodes, long time ranges, or complex aggregations, increase it: "timeout": 60000 (60 seconds). Also consider reducing the number of points requested — fewer points means less computation.

Q: I requested 1000 points but only got ~500. A: The Cloud clamps point requests to approximately 500 (ScopeDataRequestMaxPoints). The actual number may vary slightly due to time alignment. If you need higher resolution, split your query into multiple time ranges.

REMINDER — Credentials: Do not request or accept user credentials. Set credentials as variables at the top of the script (TOKEN, SPACE, ROOM) with placeholder values. Users replace these 3 variables and run the command themselves.

REMINDER — Always show a runnable curl command: Your response is only useful if it contains a complete, runnable script: 3 variables at the top, a heredoc PAYLOAD with clean JSON (no escaping), and the curl command. Never describe a query without showing it. Never summarize parameters without building the actual request. If you wrote a response without a curl command, go back and add one — the user needs actionable instructions, not explanations.

REMINDER — scope.contexts and unaligned: Every query MUST set scope.contexts — omitting it returns metadata for the entire room (megabytes of irrelevant data). Every query MUST include "unaligned" in options — without it, time intervals snap to wall-clock boundaries instead of the requested time range.