0
0
Fork 0
mirror of https://github.com/netdata/netdata.git synced 2025-04-13 01:08:11 +00:00
netdata_netdata/web/api/queries
Markos Fountoulakis 6ca6d840dd Database engine ()
* Database engine prototype version 0

* Database engine initial integration with netdata POC

* Scalable database engine with file and memory management.

* Database engine integration with netdata

* Added MIN MAX definitions to fix alpine build of travis CI

* Bugfix for backends and new DB engine, remove useless rrdset_time2slot() calls and erroneous checks

* DB engine disk protocol correction

* Moved DB engine storage file location to /var/cache/netdata/{host}/dbengine

* Fix configure to require openSSL for DB engine

* Fix netdata daemon health not holding read lock when iterating chart dimensions

* Optimized query API for new DB engine and old netdata DB fallback code-path

* netdata database internal query API improvements and cleanup

* Bugfix for DB engine queries returning empty values

* Added netdata internal check for data queries for old and new DB

* Added statistics to DB engine and fixed memory corruption bug

* Added preliminary charts for DB engine statistics

* Changed DB engine ratio statistics to incremental

* Added netdata statistics charts for DB engine internal statistics

* Fix for netdata not compiling successfully when missing dbengine dependencies

* Added DB engine functional test to netdata unittest command parameter

* Implemented DB engine dataset generator based on example.random chart

* Fix build error in CI

* Support older versions of libuv1

* Fixes segmentation fault when using multiple DB engine instances concurrently

* Fix memory corruption bug

* Fixed createdataset advanced option not exiting

* Fix for DB engine not working on FreeBSD

* Support FreeBSD library paths of new dependencies

* Workaround for unsupported O_DIRECT in OS X

* Fix unittest crashing during cleanup

* Disable DB engine FS caching in Apple OS X since O_DIRECT is not available

* Fix segfault when unittest and DB engine dataset generator don't have permissions to create temporary host

* Modified DB engine dataset generator to create multiple files

* Toned down overzealous page cache prefetcher

* Reduce internal memory fragmentation for page-cache data pages

* Added documentation describing the DB engine

* Documentation bugfixes

* Fixed unit tests compilation errors since last rebase

* Added note to back-up the DB engine files in documentation

* Added codacy fix.

* Support old gcc versions for atomic counters in DB engine
2019-05-15 08:28:06 +03:00
..
average Ga () 2018-12-07 11:30:04 +01:00
des fix incorrect use of isnormal() () 2019-03-21 19:14:14 +02:00
incremental_sum Ga () 2018-12-07 11:30:04 +01:00
max Ga () 2018-12-07 11:30:04 +01:00
median fix incorrect use of isnormal() () 2019-03-21 19:14:14 +02:00
min Ga () 2018-12-07 11:30:04 +01:00
ses fix incorrect use of isnormal() () 2019-03-21 19:14:14 +02:00
stddev fix incorrect use of isnormal() () 2019-03-21 19:14:14 +02:00
sum Ga () 2018-12-07 11:30:04 +01:00
Makefile.am fixed rpm build; () 2018-10-26 20:34:48 +03:00
query.c Database engine () 2019-05-15 08:28:06 +03:00
query.h optimized ses and added des () 2018-10-24 03:03:57 +03:00
README.md Ga () 2018-12-07 11:30:04 +01:00
rrdr.c fixed wrong annotations given to google charts () 2018-10-31 22:45:02 +02:00
rrdr.h fixed wrong annotations given to google charts () 2018-10-31 22:45:02 +02:00

Database Queries

Netdata database can be queried with /api/v1/data and /api/v1/badge.svg REST API methods.

Every data query accepts the following parameters:

name required description
chart yes The chart to be queried.
points no The number of points to be returned. Netdata can reduce number of points by applying query grouping methods. If not given, the result will have the same granularity as the database (although this relates to gtime).
before no The absolute timestamp or the relative (to now) time the query should finish evaluating data. If not given, it defaults to the timestamp of the latest point in the database.
after no The absolute timestamp or the relative (to before) time the query should start evaluating data. if not given, it defaults to the timestamp of the oldest point in the database.
group no The grouping method to use when reducing the points the database has. If not given, it defaults to average.
gtime no A resampling period to change the units of the metrics (i.e. setting this to 60 will convert per second metrics to per minute. If not given it defaults to granularity of the database.
options no A bitmap of options that can affect the operation of the query. Only 2 options are used by the query engine: unaligned and percentage. All the other options are used by the output formatters. The default is to return aligned data.
dimensions no A simple pattern to filter the dimensions to be queried. The default is to return all the dimensions of the chart.

Operation

The query engine works as follows (in this order):

Time-frame

after and before define a time-frame, accepting:

  • absolute timestamps (unix timestamps, i.e. seconds since epoch).

  • relative timestamps:

    before is relative to now and after is relative to before.

    Example: before=-60&after=-60 evaluates to the time-frame from -120 up to -60 seconds in the past, relative to the latest entry of the database of the chart.

The engine verifies that the time-frame requested is available at the database:

  • If the requested time-frame overlaps with the database, the excess requested will be truncated.

  • If the requested time-frame does not overlap with the database, the engine will return an empty data set.

At the end of this operation, after and before are absolute timestamps.

Data grouping

Database points grouping is applied when the caller requests a time-frame to be expressed with fewer points, compared to what is available at the database.

There are 2 uses that enable this feature:

  • The caller requests a specific number of points to be returned.

    For example, for a time-frame of 10 minutes, the database has 600 points (1/sec), while the caller requested these 10 minutes to be expressed in 200 points.

    This feature is used by netdata dashboards when you zoom-out the charts. The dashboard is requesting the number of points the user's screen has. This saves bandwidth and speeds up the browser (fewer points to evaluate for drawing the charts).

  • The caller requests a re-sampling of the database, by setting gtime to any value above the granularity of the chart.

    For example, the chart's units is requests/sec and caller wants requests/min.

Using points and gtime the query engine tries to find a best fit for database-points vs result-points (we call this ratio group points). It always tries to keep group points an integer. Keep in mind the query engine may shift after if required. See also the example.

Time-frame Alignment

Alignment is a very important aspect of netdata queries. Without it, the animated charts on the dashboards would constantly change shape during incremental updates.

To provide consistent grouping through time, the query engine (by default) aligns after and before to be a multiple of group points.

For example, if group points is 60 and alignment is enabled, the engine will return each point with durations XX:XX:00 - XX:XX:59, matching whole minutes.

To disable alignment, pass &options=unaligned to the query.

Query Execution

To execute the query, the engine evaluates all dimensions of the chart, one after another.

The engine does not evaluate dimensions that do not match the simple pattern given at the dimensions parameter, except when options=percentage is given (this option requires all the dimensions to be evaluated to find the percentage of each dimension vs to chart total).

For each dimension, it starts evaluating values starting at after (not inclusive) towards before (inclusive).

For each value it calls the grouping method given with the &group= query parameter (the default is average).

Grouping methods

The following grouping methods are supported. These are given all the values in the time-frame and they group the values every group points.

  • finds the minimum value
  • finds the maximum value
  • finds the average value
  • adds all the values and returns the sum
  • sorts the values and returns the value in the middle of the list
  • finds the standard deviation of the values
  • finds the relative standard deviation (coefficient of variation) of the values
  • finds the exponential weighted moving average of the values
  • applies Holt-Winters double exponential smoothing
  • finds the difference of the last vs the first value

The examples shown above, are live information from the successful web requests of the global netdata registry.

Further processing

The result of the query engine is always a structure that has dimensions and values for each dimension.

Formatting modules are then used to convert this result in many different formats and return it to the caller.

Performance

The query engine is highly optimized for speed. Most of its modules implement "online" versions of the algorithms, requiring just one pass on the database values to produce the result.

Example

When netdata is reducing metrics, it tries to return always the same boundaries. So, if we want 10s averages, it will always return points starting at a unix timestamp % 10 = 0.

Let's see why this is needed, by looking at the error case.

Assume we have 5 points:

time value
00:01 1
00:02 2
00:03 3
00:04 4
00:05 5

At 00:04 you ask for 2 points for 4 seconds in the past. So group = 2. netdata would return:

point time value
1 00:01 - 00:02 1.5
2 00:03 - 00:04 3.5

A second later the chart is to be refreshed, and makes again the same request at 00:05. These are the points that would have been returned:

point time value
1 00:02 - 00:03 2.5
2 00:04 - 00:05 4.5

Wait a moment! The chart was shifted just one point and it changed value! Point 2 was 3.5 and when shifted to point 1 is 2.5! If you see this in a chart, it's a mess. The charts change shape constantly.

For this reason, netdata always aligns the data it returns to the group.

When you request points=1, netdata understands that you need 1 point for the whole database, so group = 3600. Then it tries to find the starting point which would be timestamp % 3600 = 0 Within a database of 3600 seconds, there is one such point for sure. Then it tries to find the average of 3600 points. But, most probably it will not find 3600 of them (for just 1 out of 3600 seconds this query will return something).

So, the proper way to query the database is to also set at least after. The following call will returns 1 point for the last complete 10-second duration (it starts at timestamp % 10 = 0):

http://netdata.firehol.org/api/v1/data?chart=system.cpu&points=1&after=-10&options=seconds

When you keep calling this URL, you will see that it returns one new value every 10 seconds, and the timestamp always ends with zero. Similarly, if you say points=1&after=-5 it will always return timestamps ending with 0 or 5.

analytics