0
0
Fork 0
mirror of https://github.com/netdata/netdata.git synced 2025-04-26 22:04:46 +00:00
netdata_netdata/web/api/badges
Costa Tsaousis cb7af25c09
RRD structures managed by dictionaries ()
* rrdset - in progress

* rrdset optimal constructor; rrdset conflict

* rrdset final touches

* re-organization of rrdset object members

* prevent use-after-free

* dictionary dfe supports also counting of iterations

* rrddim managed by dictionary

* rrd.h cleanup

* DICTIONARY_ITEM now is referencing actual dictionary items in the code

* removed rrdset linked list

* Revert "removed rrdset linked list"

This reverts commit 690d6a588b4b99619c2c5e10f84e8f868ae6def5.

* removed rrdset linked list

* added comments

* Switch chart uuid to static allocation in rrdset
Remove unused functions

* rrdset_archive() and friends...

* always create rrdfamily

* enable ml_free_dimension

* rrddim_foreach done with dfe

* most custom rrddim loops replaced with rrddim_foreach

* removed accesses to rrddim->dimensions

* removed locks that are no longer needed

* rrdsetvar is now managed by the dictionary

* set rrdset is rrdsetvar, fixes https://github.com/netdata/netdata/pull/13646#issuecomment-1242574853

* conflict callback of rrdsetvar now properly checks if it has to reset the variable

* dictionary registered callbacks accept as first parameter the DICTIONARY_ITEM

* dictionary dfe now uses internal counter to report; avoided excess variables defined with dfe

* dictionary walkthrough callbacks get dictionary acquired items

* dictionary reference counters that can be dupped from zero

* added advanced functions for get and del

* rrdvar managed by dictionaries

* thread safety for rrdsetvar

* faster rrdvar initialization

* rrdvar string lengths should match in all add, del, get functions

* rrdvar internals hidden from the rest of the world

* rrdvar is now acquired throughout netdata

* hide the internal structures of rrdsetvar

* rrdsetvar is now acquired through out netdata

* rrddimvar managed by dictionary; rrddimvar linked list removed; rrddimvar structures hidden from the rest of netdata

* better error handling

* dont create variables if not initialized for health

* dont create variables if not initialized for health again

* rrdfamily is now managed by dictionaries; references of it are acquired dictionary items

* type checking on acquired objects

* rrdcalc renaming of functions

* type checking for rrdfamily_acquired

* rrdcalc managed by dictionaries

* rrdcalc double free fix

* host rrdvars is always needed

* attempt to fix deadlock 1

* attempt to fix deadlock 2

* Remove unused variable

* attempt to fix deadlock 3

* snprintfz

* rrdcalc index in rrdset fix

* Stop storing active charts and computing chart hashes

* Remove store active chart function

* Remove compute chart hash function

* Remove sql_store_chart_hash function

* Remove store_active_dimension function

* dictionary delayed destruction

* formatting and cleanup

* zero dictionary base on rrdsetvar

* added internal error to log delayed destructions of dictionaries

* typo in rrddimvar

* added debugging info to dictionary

* debug info

* fix for rrdcalc keys being empty

* remove forgotten unlock

* remove deadlock

* Switch to metadata version 5 and drop
  chart_hash
  chart_hash_map
  chart_active
  dimension_active
  v_chart_hash

* SQL cosmetic changes

* do not busy wait while destroying a referenced dictionary

* remove deadlock

* code cleanup; re-organization;

* fast cleanup and flushing of dictionaries

* number formatting fixes

* do not delete configured alerts when archiving a chart

* rrddim obsolete linked list management outside dictionaries

* removed duplicate contexts call

* fix crash when rrdfamily is not initialized

* dont keep rrddimvar referenced

* properly cleanup rrdvar

* removed some locks

* Do not attempt to cleanup chart_hash / chart_hash_map

* rrdcalctemplate managed by dictionary

* register callbacks on the right dictionary

* removed some more locks

* rrdcalc secondary index replaced with linked-list; rrdcalc labels updates are now executed by health thread

* when looking up for an alarm look using both chart id and chart name

* host initialization a bit more modular

* init rrdlabels on host update

* preparation for dictionary views

* improved comment

* unused variables without internal checks

* service threads isolation and worker info

* more worker info in service thread

* thread cancelability debugging with internal checks

* strings data races addressed; fixes https://github.com/netdata/netdata/issues/13647

* dictionary modularization

* Remove unused SQL statement definition

* unit-tested thread safety of dictionaries; removed data race conditions on dictionaries and strings; dictionaries now can detect if the caller is holds a write lock and automatically all the calls become their unsafe versions; all direct calls to unsafe version is eliminated

* remove worker_is_idle() from the exit of service functions, because we lose the lock time between loops

* rewritten dictionary to have 2 separate locks, one for indexing and another for traversal

* Update collectors/cgroups.plugin/sys_fs_cgroup.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* Update collectors/cgroups.plugin/sys_fs_cgroup.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* Update collectors/proc.plugin/proc_net_dev.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* fix memory leak in rrdset cache_dir

* minor dictionary changes

* dont use index locks in single threaded

* obsolete dict option

* rrddim options and flags separation; rrdset_done() optimization to keep array of reference pointers to rrddim;

* fix jump on uninitialized value in dictionary; remove double free of cache_dir

* addressed codacy findings

* removed debugging code

* use the private refcount on dictionaries

* make dictionary item desctructors work on dictionary destruction; strictier control on dictionary API; proper cleanup sequence on rrddim;

* more dictionary statistics

* global statistics about dictionary operations, memory, items, callbacks

* dictionary support for views - missing the public API

* removed warning about unused parameter

* chart and context name for cloud

* chart and context name for cloud, again

* dictionary statistics fixed; first implementation of dictionary views - not currently used

* only the master can globally delete an item

* context needs netdata prefix

* fix context and chart it of spins

* fix for host variables when health is not enabled

* run garbage collector on item insert too

* Fix info message; remove extra "using"

* update dict unittest for new placement of garbage collector

* we need RRDHOST->rrdvars for maintaining custom host variables

* Health initialization needs the host->host_uuid

* split STRING to its own files; no code changes other than that

* initialize health unconditionally

* unit tests do not pollute the global scope with their variables

* Skip initialization when creating archived hosts on startup. When a child connects it will initialize properly

Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
2022-09-19 23:46:13 +03:00
..
Makefile.am Makefile.am files indentation () 2019-11-11 01:30:00 +02:00
README.md Docs: fix GitHub format () 2022-04-20 07:22:51 -04:00
web_buffer_svg.c RRD structures managed by dictionaries () 2022-09-19 23:46:13 +03:00
web_buffer_svg.h netdata doubles () 2022-06-28 17:04:37 +03:00

Netdata badges

Badges are cool!

Netdata can generate badges for any chart and any dimension at any time-frame. Badges come in SVG and can be added to any web page using an <IMG> HTML tag.

Netdata badges are powerful!

Given that Netdata collects from 1.000 to 5.000 metrics per server (depending on the number of network interfaces, disks, cpu cores, applications running, users logged in, containers running, etc) and that Netdata already has data reduction/aggregation functions embedded, the badges can be quite powerful.

For each metric/dimension and for arbitrary time-frames badges can show min, max or average value, but also sum or incremental-sum to have their volume.

For example, there is a chart in Netdata that shows the current requests/s of nginx. Using this chart alone we can show the following badges (we could add more time-frames, like today, yesterday, etc):

Similarly, there is a chart that shows outbound bandwidth per class, using QoS data. So it shows kilobits/s per class. Using this chart we can show:

The right one is a volume calculation. Netdata calculated the total of the last 86.400 seconds (a day) which gives kilobits, then divided it by 8 to make it KB, then by 1024 to make it MB and then by 1024 to make it GB. Calculations like this are quite accurate, since for every value collected, every second, Netdata interpolates it to second boundary using microsecond calculations.

Let's see a few more badge examples (they come from the Netdata registry):

  • cpu usage of user root (you can pick any user; 100% = 1 core). This will be green <10%, yellow <20%, orange <50%, blue <100% (1 core), red otherwise (you define thresholds and colors on the URL).

  • mysql queries per second

    niche ones: mysql SELECT statements with JOIN, which did full table scans:


So, every single line on the charts of a Netdata dashboard, can become a badge and this badge can calculate average, min, max, or volume for any time-frame! And you can also vary the badge color using conditions on the calculated value.


How to create badges

The basic URL is http://your.netdata:19999/api/v1/badge.svg?option1&option2&option3&....

Here is what you can put for options (these are standard Netdata API options):

  • chart=CHART.NAME

    The chart to get the values from.

    This is the only parameter required and with just this parameter, Netdata will return the sum of the latest values of all chart dimensions.

    Example:

  <a href="#">
     <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu"></img>
  </a>

Which produces this:

  • alarm=NAME

    Render the current value and status of an alarm linked to the chart. This option can be ignored if the badge to be generated is not related to an alarm.

    The current value of the alarm will be rendered. The color of the badge will indicate the status of the alarm.

    For alarm badges, both chart and alarm parameters are required.

  • dimensions=DIMENSION1|DIMENSION2|...

    The dimensions of the chart to use. If you don't set any dimension, all will be used. When multiple dimensions are used, Netdata will sum their values. You can append options=absolute if you want this sum to convert all values to positive before adding them.

    Pipes in HTML have to escaped with %7C.

    Example:

  <a href="#">
     <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu&dimensions=system%7Cnice"></img>
  </a>

Which produces this:

  • before=SECONDS and after=SECONDS

    The timeframe. These can be absolute unix timestamps, or relative to now, number of seconds. By default before=0 and after=-1 (1 second in the past).

    To get the last minute set after=-60. This will give the average of the last complete minute (XX:XX:00 - XX:XX:59).

    To get the max of the last hour set after=-3600&group=max. This will give the maximum value of the last complete hour (XX:00:00 - XX:59:59)

    Example:

  <a href="#">
     <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu&after=-60"></img>
  </a>

Which produces the average of last complete minute (XX:XX:00 - XX:XX:59):

While this is the previous minute (one minute before the last one, again aligned XX:XX:00 - XX:XX:59):

  <a href="#">
     <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=system.cpu&before=-60&after=-60"></img>
  </a>

It produces this:

  • group=min or group=max or group=average (the default) or group=sum or group=incremental-sum

    If Netdata will have to reduce (aggregate) the data to calculate the value, which aggregation method to use.

    • max will find the max value for the timeframe. This works on both positive and negative dimensions. It will find the most extreme value.

    • min will find the min value for the timeframe. This works on both positive and negative dimensions. It will find the number closest to zero.

    • average will calculate the average value for the timeframe.

    • sum will sum all the values for the timeframe. This is nice for finding the volume of dimensions for a timeframe. So if you have a dimension that reports X per second, you can find the volume of the dimension in a timeframe, by adding its values in that timeframe.

    • incremental-sum will sum the difference of each value to its next. Let's assume you have a dimension that does not measure the rate of something, but the absolute value of it. So it has values like this "1, 5, 3, 7, 4". incremental-sum will calculate the difference of adjacent values. In this example, they will be (5 - 1) + (3 - 5) + (7 - 3) + (4 - 7) = 3 (which is equal to the last value minus the first = 4 - 1).

  • options=opt1|opt2|opt3|...

    These fine tune various options of the API. Here is what you can use for badges (the API has more option, but only these are useful for badges):

    • percentage, instead of returning the value, calculate the percentage of the sum of the selected dimensions, versus the sum of all the dimensions of the chart. This also sets the units to %.

    • absolute or abs, turn all values positive and then sum them.

    • display_absolute or display-absolute, to use the signed value during color calculation, but display the absolute value on the badge.

    • min2max, when multiple dimensions are given, do not sum them, but take their max - min.

    • unaligned, when data are reduced / aggregated (e.g. the request is about the average of the last minute, or hour), Netdata by default aligns them so that the charts will have a constant shape (so average per minute returns always XX:XX:00 - XX:XX:59). Setting the unaligned option, Netdata will aggregate data without any alignment, so if the request is for 60 seconds, it will aggregate the latest 60 seconds of collected data.

These are options dedicated to badges:

  • label=TEXT

    The label of the badge.

  • units=TEXT

    The units of the badge. If you want to put a /, please put a \. This is because Netdata allows badges parameters to be given as path in URL, instead of query string. You can also use null or empty to show it without any units.

    The units seconds, minutes and hours trigger special formatting. The value has to be in this unit, and Netdata will automatically change it to show a more pretty duration.

  • multiply=NUMBER

    Multiply the value with this number. The default is 1.

  • divide=NUMBER

    Divide the value with this number. The default is 1.

  • Color customization parameters

    The following parameters specify colors of each individual part of the badge. Each parameter is documented in detail below.

    Area of badge Background color parameter Text color parameter
    Label (left) part label_color text_color_lbl
    Value (right) part value_color text_color_val
    • label_color=COLOR

      The color of the label (the left part). You can use any HTML color in RGB or RRGGBB hex notation (without the # character at the beginning). Additionally, you can use one of the following predefined colors (and you can use them by their name):

      • green
      • brightgreen
      • yellow
      • yellowgreen
      • orange
      • red
      • blue
      • grey
      • gray
      • lightgrey
      • lightgray

      These colors are taken from https://github.com/badges/shields, which makes them compatible with standard badges.

    • value_color=COLOR:null|COLOR<VALUE|COLOR>VALUE|COLOR>=VALUE|COLOR<=VALUE|...

      You can add a pipe delimited list of conditions to pick the value color. The first matching (left to right) will be used.

      Example: value_color=grey:null|green<10|yellow<100|orange<1000|blue<10000|red

      The above will set grey if no value exists (not collected within the gap when lost iterations above in netdata.conf for the chart), green if the value is less than 10, yellow if the value is less than 100, and so on. Netdata will use red if no other conditions match. Only integers are supported as values.

      The supported operators are <, >, <=, >=, = (or :), and != (or <>).

      You can also use the same syntax as the label_color parameter to define each of these colors. You can reference a predefined color by name or RGB/RRGGBB hex notation.

    • text_color_lbl=RGB or text_color_lbl=RRGGBB or text_color_lbl=color_by_name

      This value specifies the font color for the font of left/label side of the badge. The syntax is the same as the label_color parameter. If not given, or given with an empty value, Netdata will use the default color.

    • text_color_val=RGB or text_color_val=RRGGBB or text_color_lbl=color_by_name

      This value specifies the font color for the font of right/value side of the badge. The syntax is the same as the label_color parameter. If not given, or given with an empty value, Netdata will use the default color.

  • precision=NUMBER

    The number of decimal digits of the value. By default Netdata will add:

    • no decimal digits for values > 1000
    • 1 decimal digit for values > 100
    • 2 decimal digits for values > 1
    • 3 decimal digits for values > 0.1
    • 4 decimal digits for values <= 0.1

    Using the precision=NUMBER you can set your preference per badge.

  • scale=XXX

    This option scales the svg image. It accepts values above or equal to 100 (100% is the default scale). For example, lets get a few different sizes:

    original
    scale=125
    scale=150
    scale=175
    scale=200

  • fixed_width_lbl=NUMBER and fixed_width_val=NUMBER

    This parameter overrides auto-sizing of badges and displays them at fixed widths. fixed_width_lbl determines the size of the label's left side (label/name). fixed_width_val determines the size of the the label's right side (value). You must set both parameters together, or they will be ignored.

    You should set the label/value widths wide enough to provide space for all the possible values/contents of the badge you're requesting. In case the text cannot fit the space given it will be clipped.

    The scale parameter still applies on the values you give to fixed_width_lbl and fixed_width_val.

  • refresh=auto or refresh=SECONDS

    This option enables auto-refreshing of images. Netdata will send the HTTP header Refresh: SECONDS to the web browser, thus requesting automatic refresh of the images at regular intervals.

    auto will calculate the proper SECONDS to avoid unnecessary refreshes. If SECONDS is zero, this feature is disabled (it is also disabled by default).

    Auto-refreshing like this, works only if you access the badge directly. So, you may have to put it an embed or iframe for it to be auto-refreshed. Use something like this:

<embed src="BADGE_URL" type="image/svg+xml" height="20" />

Another way is to use javascript to auto-refresh them. You can auto-refresh all the Netdata badges on a page using javascript. You have to add a class to all the Netdata badges, like this <img class="netdata-badge" src="..."/>. Then add this javascript code to your page (it requires jquery):

<script>
    var NETDATA_BADGES_AUTOREFRESH_SECONDS = 5;
    function refreshNetdataBadges() {
      var now = new Date().getTime().toString();
      $('.netdata-badge').each(function() {
        this.src = this.src.replace(/\&_=\d*/, '') + '&_=' + now;
      });
      setTimeout(refreshNetdataBadges, NETDATA_BADGES_AUTOREFRESH_SECONDS * 1000);
    }
    setTimeout(refreshNetdataBadges, NETDATA_BADGES_AUTOREFRESH_SECONDS * 1000);
</script>

A more advanced badges refresh method is to include http://your.netdata.ip:19999/refresh-badges.js in your page.


Escaping URLs

Keep in mind that if you add badge URLs to your HTML pages you have to escape the special characters:

character name escape sequence
`` space (in labels and units) %20
# hash (for colors) %23
% percent (in units) %25
< less than %3C
> greater than %3E
\ backslash (when you need a /) %5C
| pipe (delimiting parameters) %7C

FAQ

Is it fast?

On modern hardware, Netdata can generate about 2.000 badges per second per core, before noticing any delays. It generates a badge in about half a millisecond!

Of course these timing are for badges that use recent data. If you need badges that do calculations over long durations (a day, or more), timing will differ. Netdata logs its timings at its access.log, so take a look there before adding a heavy badge on a busy web site. Of course, you can cache such badges or have a cron job get them from Netdata and save them at your web server at regular intervals.

Embedding badges in GitHub

You have 2 options:

  • SVG images with markdown
  • SVG images with HTML (directly in .md files)

For example, this is the cpu badge shown above:

  • Markdown example:
[![A nice name](https://registry.my-netdata.io/api/v1/badge.svg?chart=users.cpu&dimensions=root&value_color=grey:null%7Cgreen%3C10%7Cyellow%3C20%7Corange%3C50%7Cblue%3C100%7Cred&label=root%20user%20cpu%20now&units=%25)](https://registry.my-netdata.io/#apps_cpu)
  • HTML example:
<a href="https://registry.my-netdata.io/#apps_cpu">
    <img src="https://registry.my-netdata.io/api/v1/badge.svg?chart=users.cpu&dimensions=root&value_color=grey:null%7Cgreen%3C10%7Cyellow%3C20%7Corange%3C50%7Cblue%3C100%7Cred&label=root%20user%20cpu%20now&units=%25"></img>
</a>

Both produce this:

Auto-refreshing badges in GitHub

Unfortunately it cannot be done. GitHub fetches all the images using a proxy and rewrites all the URLs to be served by the proxy.

You can refresh them from your browser console though. Press F12 to open the web browser console (switch to the console too), paste the following and press enter. They will refresh:

var len = document.images.length; while(len--) { document.images[len].src = document.images[len].src.replace(/\?cacheBuster=\d*/, "") + "?cacheBuster=" + new Date().getTime().toString(); };