updated queries README (#4484)
Costa Tsaousis committed
Oct 25, 2018 at 04:08 UTC
5d1feb195a29b7baccf7b3a4f3aa70111d5a41b1
1 file changed
+90
-86
web/api/queries/README.md
+90
-86
@@ -1,111 +1,117 @@
1
# Database Queries
2
3
-Netdata database can be queried with `/api/v1/data` and `/api/v1/badge.svg` API methods.
3
+Netdata database can be queried with `/api/v1/data` and `/api/v1/badge.svg` REST API methods.
4
5
Every data query accepts the following parameters:
6
7
-name|description
8
-:----:|:----:
9
-`chart`|The chart to be queried.
10
-`points`|The number of points to be returned. Netdata can reduce number of points by applying query grouping methods.
11
-`before`|The absolute timestamp or the relative (to now) time the query should finish evaluating data.
12
-`after`|The absolute timestamp or the relative (to `before`) time the query should start evaluating data.
13
-`group`|The grouping method to use when reducing the points the database has.
14
-`gtime`|A resampling period to change the units of the metrics (i.e. setting this to `60` will convert `per second` metrics to `per minute`.
15
-`options`|A bitmap of options that can affect the operation of the query. Only 2 options are used by the query engine: `unaligned` and `percentage`. All the other options are used by the output formatters.
16
-`dimensions`|A simple pattern to filter the dimensions to be queried.
7
+name|required|description
8
+:----:|:----:|:---
9
+`chart`|yes|The chart to be queried.
10
+`points`|no|The number of points to be returned. Netdata can reduce number of points by applying query grouping methods. If not given, the result will have the same granularity as the database (although this relates to `gtime`).
11
+`before`|no|The absolute timestamp or the relative (to now) time the query should finish evaluating data. If not given, it defaults to the timestamp of the latest point in the database.
12
+`after`|no|The absolute timestamp or the relative (to `before`) time the query should start evaluating data. if not given, it defaults to the timestamp of the oldest point in the database.
13
+`group`|no|The grouping method to use when reducing the points the database has. If not given, it defaults to `average`.
14
+`gtime`|no|A resampling period to change the units of the metrics (i.e. setting this to `60` will convert `per second` metrics to `per minute`. If not given it defaults to granularity of the database.
15
+`options`|no|A bitmap of options that can affect the operation of the query. Only 2 options are used by the query engine: `unaligned` and `percentage`. All the other options are used by the output formatters. The default is to return aligned data.
16
+`dimensions`|no|A simple pattern to filter the dimensions to be queried. The default is to return all the dimensions of the chart.
17
18
## Operation
19
20
The query engine works as follows (in this order):
21
22
-1. **Identify the exact time-frame required, in absolute timestamps.**
22
+#### Time-frame
23
24
- `after` and `before` define a time-frame:
25
-
26
- - in **absolute timestamps** (unix timestamps, i.e. seconds since epoch).
27
-
28
- - in **relative timestamps**:
29
-
30
- `before` is relative to now and `after` is relative to `before`.
31
-
32
- So, `before=-60&after=-60` evaluates to the time-frame from -120 up to -60 seconds in
33
- the past, relative to now.
24
+`after` and `before` define a time-frame, accepting:
25
35
- At the end of this operation, `after` and `before` are absolute timestamps.
36
- The engine verifies that the time-frame is available at the database. If it is not,
37
- it will adjust `after` and `before` accordingly so that usable data can be returned,
38
- or no data at all if the time-frame is entirely outside the current range of the
39
- database.
26
+- **absolute timestamps** (unix timestamps, i.e. seconds since epoch).
27
41
-2. **Identify the grouping of database points required.**
28
+- **relative timestamps**:
29
43
- Grouping database points is used when the caller requests a longer time-frame to be
44
- expressed with fewer points, compared to what is available at the database.
45
-
46
- There are 2 uses of this (that can be combined):
47
-
48
- - The caller requests a specific number of `points` to be returned.
49
-
50
- For example, for a time-frame of 10 minutes, the database has 600 points (1/sec),
51
- while the caller requested these 10 minutes to be expressed in 200 points.
52
-
53
- This feature is used by netdata dashboards when you zoom-out the charts.
54
- The dashboard is requesting the number of points the user's screen has, and netdata
55
- returns that many points to perfectly match the screen. This saves bandwidth
56
- and makes drawing the charts a lot faster.
57
-
58
- - The caller requests a **re-sampling** of the database, by setting `gtime` to any value
59
- above `1`. For example, the database maintains the metrics in the form of `X/sec`
60
- but the caller set `gtime=60` to get `X/min`.
61
-
62
- Using the above information the query engine tries to find a best fit for database-points
63
- to result-points ratio (we call this `group points`). It always tries to keep `group points`
64
- an integer. Keep in mind the query engine may alter a bit `after` if required. So, the engine
65
- may decide to shift the starting point of the time-frame to keep the query optimal.
66
-
67
-3. **Align the time-frame.**
30
+ `before` is relative to now and `after` is relative to `before`.
31
+
32
+ Example: `before=-60&after=-60` evaluates to the time-frame from -120 up to -60 seconds in
33
+ the past, relative to the latest entry of the database of the chart.
34
69
- Alignment is a very important aspect of netdata queries. Without it, the animated
70
- charts on the dashboards would constantly change shape during incremental updates.
71
- To provide consistent grouping of all points, the query engine (by default) aligns
72
- `after` and `before` to be a multiple of `group points`.
73
-
74
- For example, if `group points` is 60 and alignment is enabled, the engine will return
75
- each point with durations XX:XX:00 - XX:XX:59 matching minutes. Of course, depending
76
- on the database granularity for the specific chart and the requested points to be
77
- returned, the engine may use any integer number for `group points`.
78
-
79
- To disable alignment, pass `&options=unaligned` to the query.
35
+The engine verifies that the time-frame requested is available at the database:
36
+
37
+- If the requested time-frame overlaps with the database, the excess requested
38
+ will be truncated.
39
81
-4. **Execute the query**
40
+- If the requested time-frame does not overlap with the database, the engine will
41
+ return an empty data set.
42
+
43
+At the end of this operation, `after` and `before` are absolute timestamps.
44
+
45
+#### Data grouping
46
+
47
+Database points grouping is applied when the caller requests a time-frame to be
48
+expressed with fewer points, compared to what is available at the database.
49
83
- To execute the query, the engine evaluates all dimensions of the chart, one after another.
84
- The engine will not evaluate dimensions that do not match the simple pattern given at
85
- the `dimensions` parameter, except when `options=percentage` is given (this option requires
86
- all the dimensions to be evaluated to find the percentage of each dimension vs to chart
87
- total).
50
+There are 2 uses that enable this feature:
51
+
52
+- The caller requests a specific number of `points` to be returned.
53
+
54
+ For example, for a time-frame of 10 minutes, the database has 600 points (1/sec),
55
+ while the caller requested these 10 minutes to be expressed in 200 points.
56
+
57
+ This feature is used by netdata dashboards when you zoom-out the charts.
58
+ The dashboard is requesting the number of points the user's screen has.
59
+ This saves bandwidth and speeds up the browser (fewer points to evaluate for drawing the charts).
60
+
61
+- The caller requests a **re-sampling** of the database, by setting `gtime` to any value
62
+ above the granularity of the chart.
63
+
64
+ For example, the chart's units is `requests/sec` and caller wants `requests/min`.
65
+
66
+Using `points` and `gtime` the query engine tries to find a best fit for **database-points**
67
+vs **result-points** (we call this ratio `group points`). It always tries to keep `group points`
68
+an integer. Keep in mind the query engine may shift `after` if required.
69
+
70
+#### Time-frame Alignment
71
+
72
+Alignment is a very important aspect of netdata queries. Without it, the animated
73
+charts on the dashboards would constantly change shape during incremental updates.
74
+
75
+To provide consistent grouping through time, the query engine (by default) aligns
76
+`after` and `before` to be a multiple of `group points`.
77
+
78
+For example, if `group points` is 60 and alignment is enabled, the engine will return
79
+each point with durations XX:XX:00 - XX:XX:59, matching whole minutes.
80
+
81
+To disable alignment, pass `&options=unaligned` to the query.
82
89
- For each dimension, it starts evaluating values from `after` towards `before`.
90
- For each value it calls the **grouping method** specified (the default is `average`).
83
+#### Query Execution
84
+
85
+To execute the query, the engine evaluates all dimensions of the chart, one after another.
86
+
87
+The engine does not evaluate dimensions that do not match the [simple pattern](../../../libnetdata/simple_pattern)
88
+given at the `dimensions` parameter, except when `options=percentage` is given (this option
89
+requires all the dimensions to be evaluated to find the percentage of each dimension vs to chart
90
+total).
91
+
92
+For each dimension, it starts evaluating values starting at `after` (not inclusive) towards
93
+`before` (inclusive).
94
+
95
+For each value it calls the **grouping method** given with the `&group=` query parameter
96
+(the default is `average`).
97
98
## Grouping methods
99
100
The following grouping methods are supported. These are given all the values in the time-frame
101
and they group the values every `group points`.
102
97
-name|identifier(s)|description
98
-:---:|:---:|:---:
99
-Min|`min`|finds the minimum value
100
-Max|`max`|finds the maximum value
101
-Average|`average` `mean`|finds the average value
102
-Sum|`sum`|adds all the values and returns the sum
103
-Median|`median`|sorts the values and returns the value in the middle of the list
104
-Standard Deviation|`stddev`|finds the standard deviation of the values
105
-Coefficient of Variation|`cv` `rds`|finds the relative standard deviation of the values
106
-Single Exponential Smoothing|`ses` `ema` `ewma`|finds the exponential weighted moving average of the values
107
-Double Exponential Smoothing|`des`|applies Holt-Winters double exponential smoothing
108
-Incremental Sum|`incremental_sum` `incremental-sum`|find the difference of the last vs the first values
103
+Name|Identifier(s)|Description|Live Example
104
+:---:|:---:|:---:|:---
105
+Min|`min`|finds the minimum value|
106
+Max|`max`|finds the maximum value|
107
+Average|`average`, `mean`|finds the average value|
108
+Sum|`sum`|adds all the values and returns the sum|
109
+Median|`median`|sorts the values and returns the value in the middle of the list|
110
+Standard Deviation|`stddev`|finds the standard deviation of the values|
111
+Coefficient of Variation|`cv`, `rds`|finds the relative standard deviation of the values|
112
+Single Exponential Smoothing|`ses`, `ema`, `ewma`|finds the exponential weighted moving average of the values|
113
+Double Exponential Smoothing|`des`|applies Holt-Winters double exponential smoothing|
114
+Incremental Sum|`incremental_sum`, `incremental-sum`|find the difference of the last vs the first values|
115
116
## Further processing
117
@@ -120,5 +126,3 @@ to the caller.
126
The query engine is highly optimized for speed. Most of its modules implement "online"
127
versions of the algorithms, requiring just one pass on the database values to produce
128
the result.
123
-
124
-