49234f23de
* Tier part 1 * Tier part 2 * Tier part 3 * Tier part 4 * Tier part 5 * Fix some ML compilation errors * fix more conflicts * pass proper tier * move metric_uuid from state to RRDDIM * move aclk_live_status from state to RRDDIM * move ml_dimension from state to RRDDIM * abstracted the data collection interface * support flushing for mem db too * abstracted the query api * abstracted latest/oldest time per metric * cleanup * store_metric for tier1 * fix for store_metric * allow multiple tiers, more than 2 * state to tier * Change storage type in db. Query param to request min, max, sum or average * Store tier data correctly * Fix skipping tier page type * Add tier grouping in the tier * Fix to handle archived charts (part 1) * Temp fix for query granularity when requesting tier1 data * Fix parameters in the correct order and calculate the anomaly based on the anomaly count * Proper tiering grouping * Anomaly calculation based on anomaly count * force type checking on storage handles * update cmocka tests * fully dynamic number of storage tiers * fix static allocation * configure grouping for all tiers; disable tiers for unittest; disable statsd configuration for private charts mode * use default page dt using the tiering info * automatic selection of tier * fix for automatic selection of tier * working prototype of dynamic tier selection * automatic selection of tier done right (I hope) * ask for the proper tier value, based on the grouping function * fixes for unittests and load_metric_next() * fixes for lgtm findings * minor renames * add dbengine to page cache size setting * add dbengine to page cache with malloc * query engine optimized to loop as little are required based on the view_update_every * query engine grouping methods now do not assume a constant number of points per group and they allocate memory with OWA * report db points per tier in jsonwrap * query planer that switches database tiers on the fly to satisfy the query for the entire timeframe * dbegnine statistics and documentation (in progress) * calculate average point duration in db * handle single point pages the best we can * handle single point pages even better * Keep page type in the rrdeng_page_descr * updated doc * handle future backwards compatibility - improved statistics * support &tier=X in queries * enfore increasing iterations on tiers * tier 1 is always 1 iteration * backfilling higher tiers on first data collection * reversed anomaly bit * set up to 5 tiers * natural points should only be offered on tier 0, except a specific tier is selected * do not allow more than 65535 points of tier0 to be aggregated on any tier * Work only on actually activated tiers * fix query interpolation * fix query interpolation again * fix lgtm finding * Activate one tier for now * backfilling of higher tiers using raw metrics from lower tiers * fix for crash on start when storage tiers is increased from the default * more statistics on exit * fix bug that prevented higher tiers to get any values; added backfilling options * fixed the statistics log line * removed limit of 255 iterations per tier; moved the code of freezing rd->tiers[x]->db_metric_handle * fixed division by zero on zero points_wanted * removed dead code * Decide on the descr->type for the type of metric * dont store metrics on unknown page types * free db_metric_handle on sql based context queries * Disable STORAGE_POINT value check in the exporting engine unit tests * fix for db modes other than dbengine * fix for aclk archived chart queries destroying db_metric_handles of valid rrddims * fix left-over freez() instead of OWA freez on median queries Co-authored-by: Costa Tsaousis <costa@netdata.cloud> Co-authored-by: Vladimir Kobal <vlad@prokk.net> |
||
|---|---|---|
| .. | ||
| average | ||
| countif | ||
| des | ||
| incremental_sum | ||
| max | ||
| median | ||
| min | ||
| ses | ||
| stddev | ||
| sum | ||
| Makefile.am | ||
| query.c | ||
| query.h | ||
| README.md | ||
| rrdr.c | ||
| rrdr.h | ||
Database Queries
Netdata database can be queried with /api/v1/data and /api/v1/badge.svg REST API methods.
Every data query accepts the following parameters:
| name | required | description |
|---|---|---|
chart |
yes | The chart to be queried. |
points |
no | The number of points to be returned. Netdata can reduce number of points by applying query grouping methods. If not given, the result will have the same granularity as the database (although this relates to gtime). |
before |
no | The absolute timestamp or the relative (to now) time the query should finish evaluating data. If not given, it defaults to the timestamp of the latest point in the database. |
after |
no | The absolute timestamp or the relative (to before) time the query should start evaluating data. if not given, it defaults to the timestamp of the oldest point in the database. |
group |
no | The grouping method to use when reducing the points the database has. If not given, it defaults to average. |
gtime |
no | A resampling period to change the units of the metrics (i.e. setting this to 60 will convert per second metrics to per minute. If not given it defaults to granularity of the database. |
options |
no | A bitmap of options that can affect the operation of the query. Only 2 options are used by the query engine: unaligned and percentage. All the other options are used by the output formatters. The default is to return aligned data. |
dimensions |
no | A simple pattern to filter the dimensions to be queried. The default is to return all the dimensions of the chart. |
Operation
The query engine works as follows (in this order):
Time-frame
after and before define a time-frame, accepting:
-
absolute timestamps (unix timestamps, i.e. seconds since epoch).
-
relative timestamps:
beforeis relative to now andafteris relative tobefore.Example:
before=-60&after=-60evaluates to the time-frame from -120 up to -60 seconds in the past, relative to the latest entry of the database of the chart.
The engine verifies that the time-frame requested is available at the database:
-
If the requested time-frame overlaps with the database, the excess requested will be truncated.
-
If the requested time-frame does not overlap with the database, the engine will return an empty data set.
At the end of this operation, after and before are absolute timestamps.
Data grouping
Database points grouping is applied when the caller requests a time-frame to be expressed with fewer points, compared to what is available at the database.
There are 2 uses that enable this feature:
-
The caller requests a specific number of
pointsto be returned.For example, for a time-frame of 10 minutes, the database has 600 points (1/sec), while the caller requested these 10 minutes to be expressed in 200 points.
This feature is used by Netdata dashboards when you zoom-out the charts. The dashboard is requesting the number of points the user's screen has. This saves bandwidth and speeds up the browser (fewer points to evaluate for drawing the charts).
-
The caller requests a re-sampling of the database, by setting
gtimeto any value above the granularity of the chart.For example, the chart's units is
requests/secand caller wantsrequests/min.
Using points and gtime the query engine tries to find a best fit for database-points
vs result-points (we call this ratio group points). It always tries to keep group points
an integer. Keep in mind the query engine may shift after if required. See also the example.
Time-frame Alignment
Alignment is a very important aspect of Netdata queries. Without it, the animated charts on the dashboards would constantly change shape during incremental updates.
To provide consistent grouping through time, the query engine (by default) aligns
after and before to be a multiple of group points.
For example, if group points is 60 and alignment is enabled, the engine will return
each point with durations XX:XX:00 - XX:XX:59, matching whole minutes.
To disable alignment, pass &options=unaligned to the query.
Query Execution
To execute the query, the engine evaluates all dimensions of the chart, one after another.
The engine does not evaluate dimensions that do not match the simple pattern
given at the dimensions parameter, except when options=percentage is given (this option
requires all the dimensions to be evaluated to find the percentage of each dimension vs to chart
total).
For each dimension, it starts evaluating values starting at after (not inclusive) towards
before (inclusive).
For each value it calls the grouping method given with the &group= query parameter
(the default is average).
Grouping methods
The following grouping methods are supported. These are given all the values in the time-frame
and they group the values every group points.
finds the minimum value
finds the maximum value
finds the average value
adds all the values and returns the sum
sorts the values and returns the value in the middle of the list
finds the standard deviation of the values
finds the relative standard deviation (coefficient of variation) of the values
finds the exponential weighted moving average of the values
applies Holt-Winters double exponential smoothing
finds the difference of the last vs the first value
The examples shown above, are live information from the successful web requests of the global Netdata registry.
Further processing
The result of the query engine is always a structure that has dimensions and values for each dimension.
Formatting modules are then used to convert this result in many different formats and return it to the caller.
Performance
The query engine is highly optimized for speed. Most of its modules implement "online" versions of the algorithms, requiring just one pass on the database values to produce the result.
Example
When Netdata is reducing metrics, it tries to return always the same boundaries. So, if we want 10s averages, it will always return points starting at a unix timestamp % 10 = 0.
Let's see why this is needed, by looking at the error case.
Assume we have 5 points:
| time | value |
|---|---|
| 00:01 | 1 |
| 00:02 | 2 |
| 00:03 | 3 |
| 00:04 | 4 |
| 00:05 | 5 |
At 00:04 you ask for 2 points for 4 seconds in the past. So group = 2. Netdata would return:
| point | time | value |
|---|---|---|
| 1 | 00:01 - 00:02 | 1.5 |
| 2 | 00:03 - 00:04 | 3.5 |
A second later the chart is to be refreshed, and makes again the same request at 00:05. These are the points that would have been returned:
| point | time | value |
|---|---|---|
| 1 | 00:02 - 00:03 | 2.5 |
| 2 | 00:04 - 00:05 | 4.5 |
Wait a moment! The chart was shifted just one point and it changed value! Point 2 was 3.5 and when shifted to point 1 is 2.5! If you see this in a chart, it's a mess. The charts change shape constantly.
For this reason, Netdata always aligns the data it returns to the group.
When you request points=1, Netdata understands that you need 1 point for the whole database, so group = 3600. Then it tries to find the starting point which would be timestamp % 3600 = 0 Within a database of 3600 seconds, there is one such point for sure. Then it tries to find the average of 3600 points. But, most probably it will not find 3600 of them (for just 1 out of 3600 seconds this query will return something).
So, the proper way to query the database is to also set at least after. The following call will returns 1 point for the last complete 10-second duration (it starts at timestamp % 10 = 0):
http://netdata.firehol.org/api/v1/data?chart=system.cpu&points=1&after=-10&options=seconds
When you keep calling this URL, you will see that it returns one new value every 10 seconds, and the timestamp always ends with zero. Similarly, if you say points=1&after=-5 it will always return timestamps ending with 0 or 5.